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ABSTRACT 


Discrete  Event  Simulation  (DES)  is  one  of  the  most  widely  used  methodologies 
for  Operations  Research  (OR)  modeling  and  analysis.  However,  designing  and 
implementing  DES  ean  be  a  time-consuming  and  error-prone  task.  This  thesis  designed, 
implemented  and  evaluated  a  tool,  the  Event  Graph  Graphieal  Design  Tool  (EGGDT),  to 
help  OR  analysts  in  the  design,  implementation,  and  maintenanee  of  DES  reducing  the 
development  and  debugging  times. 

The  Unified  Modeling  Eanguage  was  used  to  doeument  the  development  of  the 
EGGDT,  which  was  programmed  in  Java  using  J2D  and  Swing.  Human  Eactors 
teehniques  were  employed  to  help  in  the  design  proeess  and  to  evaluate  the  final 
prototype  of  the  EGGDT. 

During  the  design  proeess,  two  formative  experiments  were  performed  to  evaluate 
the  Graphieal  User  Interfaee  design  decisions.  A  final  summative  experiment  was  done  to 
test  if  the  potential  users  consider  the  tool  a  useful  means  to  develop  OR  simulations. 
Partieipants  of  the  experiments  agreed  that  tools  like  the  EGGDT  are  an  essential 
instrument  when  developing  simulations. 
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DISCLAIMER 


The  reader  is  cautioned  that  computer  programs  developed  in  this  research  may 
not  have  been  exercised  for  all  cases  of  interest.  While  every  effort  has  been  made,  within 
the  time  available,  to  ensure  that  the  programs  are  free  of  computational  and  logic  errors, 
they  cannot  be  considered  validated.  Any  application  of  these  programs  without 
additional  verification  is  at  risk  of  the  planner. 
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EXECUTIVE  SUMMARY 


In  a  constantly  changing  world,  decision-makers  faee  problems  with  high  levels 
of  uneertainty.  Operations  Researeh  (OR)  analysts  help  by  seientifieally  studying  the 
different  alternatives  for  each  problem  and  proposing  solutions.  Among  the  many 
teehniques  used  by  OR  analysts,  simulation  is  one  of  the  most  important. 

Easy-to-use  applieations  to  design  simulations  models,  like  Arena,  ean  be  found. 
These  applieations  elaim  they  provide  a  visual  environment  to  model  simulation  and  a 
press-a-button  analysis  of  the  simulation  outeomes.  However,  these  applieations  eannot 
solve  all  kinds  of  problems  and,  even  worse,  they  are  not  sealable. 

A  more  flexible  teehnique  to  implement  simulation  models  is  to  use  Event  Graphs 
to  deseribe  the  Diserete  Event  behavior  of  the  systems  and  to  generate  a  eomputer 
program  based  on  this  model.  This  approaeh  has  the  advantage  of  being  both  flexible  and 
sealable.  Many  systems  may  be  modeled  using  disere te-event  simulation,  ineluding 
produetion  systems,  transport  systems,  weapons  systems,  or  military  operations. 

During  this  thesis  researeh  an  Event  Graph  Graphieal  Design  Tool  (EGGDT)  was 
developed  to  help  OR  analysts  in  designing  EGs.  The  EGGDT  provides  a  computer 
environment  to  draw  EG  elements,  to  define  simulation  variables  and  to  generate  the 
skeleton  of  the  Java  souree  eode  of  the  simulation  using  Simkit.  The  EGGDT  ean  reduce 
simulation  development  and  debugging  times. 

Eor  the  Analysis  and  Design  (A&D)  phases  of  the  EGGDT,  the  Unified  Modeling 
Eanguage  (UME)  was  use.  The  UME  allowed  the  depiction  of  the  A&D  decisions  and 
was  used  to  doeument  the  applieation. 

Sinee  the  EGGDT  is  a  graphieal  applieation  to  be  used  by  OR  analysts,  it  was 
necessary  to  consider  the  Human  Eaetors  involved  in  its  development.  The  User-Centered 
approaeh,  used  to  develop  the  EGGDT,  is  when  the  final  user  is  considered  and  eonsulted 
for  every  major  deeision  in  every  phase  of  the  development  of  the  system.  The  final  users 
of  the  EGGDT  are  OR  students  at  the  Naval  Postgraduate  School  (NPS).  Three 


XIX 


experiments  were  performed  as  a  part  of  the  effort  to  involve  the  user  in  the  development 
of  the  Graphieal  User  Interfaee  (GUI)  of  the  EGGDT. 

The  first  experiment  tested  the  users’  opinion  about  the  initial  design  of  the  GUI 
of  the  EGGDT.  The  prototype  used  for  the  experiment  implemented  all  the  interfaee 
serviees  required  to  the  EGGDT  but  did  not  provided  any  funetionality.  The  reaetion  of 
the  partieipants  in  the  experiment  was  positive,  so  the  general  layout  of  the  GUI  was 
eonsidered  adequate  and  used  in  following  prototypes  and  in  the  final  version  of  the 
EGGDT.  However,  the  partieipants  expressed  difficulties  in  creating  the  edges  of  the 
EGs;  in  an  EG,  circles  represent  events,  while  arcs  represent  edges. 

The  second  experiment  focused  on  the  edge  construction.  Two  methods  were 
proposed  and  considered  by  the  users.  Participants  in  the  experiment  expressed  their 
preference  for  the  “Tree  Method”  that  allowed  the  drawing  of  edges  by  specifying  the  end 
events  and  any  number  of  middle  points.  The  experiment  also  showed  that  participants 
did  their  tasks  faster  with  the  “Tree  Method”  than  they  did  with  the  other  one.  The  “Tree 
Method”  was  then  selected  to  implement  the  EGGDT. 

To  test  the  claim  that  simulation-design  tools  help  OR  analysts  who  are 
developing  simulation  models,  a  final  experiment  was  performed.  Participants  in  the 
experiment  agreed  that  tools  like  the  EGGDT  could  improve  their  satisfaction  in 
developing  simulations;  they  also  unanimously  stated  their  preference  for  using  these 
tools  over  manual  methods.  In  addition,  they  expressed  their  opinion  that  these  tools 
could  be  useful  in  a  wide  range  of  OR  applications.  Einally,  they  stated  that  tools  like  the 
EGGDT  encouraged  them  to  have  more  confidence  when  facing  simulation  projects. 

In  conclusion,  this  investigation  shows  that  OR  students  at  the  NPS  consider  the 
EGGDT  and  similar  tools  an  essential  instrument  when  developing  simulations.  The 
principal  investigator  also  believes  that  this  statement  can  be  generalized  to  all  OR 
analysts  and  to  others  involved  in  modeling  and  simulation  projects  and  studies. 
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UME-CD . UME  Class  Diagram 
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I.  INTRODUCTION 


A,  AREA  OF  RESEARCH 

One  of  the  major  tools  for  Operations  Researeh  (OR)  is  simulation.  This 
teehnique  is  used  in  many  applieations,  sueh  as  reliability,  aequisition  planning, 
transportation,  and  system  assessment.  The  value  of  simulation  is  evident  by  the  amount 
of  money  the  US  Department  of  Defense  is  expending  on  the  Joint  Warfare  System 
(JWARS)  project.  JWARS  will  be  a  campaign-level  simulation  model  providing  a 
simulation  of  joint  warfare  that  will  support  “operational  planning  and  execution,  force 
assessment  studies,  system  trade  analyses,  and  concept  and  doctrine  developmenf’ 
[MAXOO]. 

Analysts  in  OR  implement  simulations  on  computers  using  computer  software 
applications  or  programming  languages.  Consequently,  the  quality  of  the  simulation  - 
and  of  the  whole  OR  project  as  well  -  depends  on  the  quality  of  its  software.  Computer 
Aided  Software  Engineering  (CASE)  tools  can  help  software  engineers  to  develop 
computer  applications,  they  can  also  help  OR  analysts  to  model  simulations.  In  designing 
CASE  tools,  the  area  of  Human  Eactors  must  be  considered.  These  tools  are  intended  for 
use  by  OR  analysts;  therefore,  the  productivity  achieved  depends  on  their  usability  and  on 
the  level  at  which  they  are  accepted  by  the  user.  This  thesis  designs  and  implements  one 
type  of  CASE  tool  and  evaluates  the  human  factors  acceptability  of  this  tool. 

B,  BACKGROUND 

1,  Simulation  Development 

a.  Discrete  Event  Simulation  and  Event  Graphs 

The  Discrete -Event-Simulation  (DES)  paradigm  is  the  preferred 
framework  for  OR  simulations.  Many  systems  in  OR  studies  can  be  modeled  as  discrete- 
event  systems.  Eor  example,  DES  can  be  used  to  model  production  systems,  transport 
systems,  weapons  systems,  or  military  operations.  Discrete-event  systems  are  those 
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whose  state  ehanges  over  time.  Sueh  systems  are  defined  by  piecewise  constant  state 
trajectories. 


Event  Graphs  (EG),  introduced  by  Schruben  [SCH83],  are  practical  means 
of  representing  DES  models.  These  minimalist  graphs  allow  depiction  of  the  behavior  of 
the  system.  Only  four  elements  exist  in  an  EG  (see  [BEISOI]): 

•  Simulation  parameters  represent  the  characteristics  of  the  system,  for 
example,  the  random  variable  “Arrival  Time”  or  the  system’s  constant 
“Maximum  Number  of  Servers”. 

•  State  variables  convey  the  state  of  the  system.  DES  systems  are 
studied  by  tracking  the  changes  of  the  values  of  these  variables.  Some 
examples  of  state  variables  are  “Number  of  Available  Servers”  over 
time  or  the  “Total  Number  of  Customer  Served”. 

•  Events  represent  a  particular  state  transition  in  the  system.  When  an 
event  is  fired  by  the  simulation’s  controller,  its  actions  are  executed 
and  the  events  specified  by  its  outgoing  edges  can  be  scheduled  (an 
event  can  only  be  scheduled  by  another  event).  Examples  of  events  are 
the  “Arrival”  event  or  the  “Start  Service”  event. 

•  Scheduling  Edges  represent  the  scheduling  of  one  event  to  distinguish 
it  from  another.  This  scheduling  can  be  restricted  by  a  delay  time  and  a 
condition.  Eor  example,  the  event  “Arrival”  schedules  the  event  “Start 
Service”  if  the  state  variable  “Number  of  Available  Servers”  is  greater 
than  zero,  that  is,  if  a  server  is  available. 

The  simulation  time  and  event  flow  are  governed  by  the  event  list.  Using 
this  list,  the  simulation’s  controller  determines  which  event  to  fire. 

b.  Simkit 

Simkit  was  first  implemented  in  a  master’s  thesis  by  ET  Kirk  Stork,  US 
Navy  [ST096]  and  has  subsequently  been  reviewed  and  extended  by  Professor  Arnold 
Buss.  Since  1997,  Simkit  has  been  used  to  teach  DES  in  the  OR  Department  at  NPS. 
Simkit  is  a  simulation  engine  implemented  as  a  Java  library  supporting  the  realization  of 
DES  models.  Simkit  provides  methods  to  run  iterations,  control  parameters  of 
experiments,  and  gather  output  data. 

EGs  and  Simkit  are  utilized  to  graphically  design  and  implement  DES 
models.  They  have  a  straightforward  correspondence;  that  is,  for  every  element  in  an  EG 
there  is  specific  Java  code  that  Simkit  interprets. 
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Elements  of  an  EG  and  Simkit  assoeiated  eode  are  given  in  Table  I-l.  See 
Appendix  A  for  details  and  a  simple  example 


EG 

Simkit 

State  Variables 

Class  instance  variable 

Simulation  Parameters 

Class  instance  variable 

Events 

Class  “do”  method 

Event  Parameters 

Parameter  of  a  “do”  method 

Event  Actions 

Code  lines  of  a  “do”  method 

Scheduling  Edges 

“waitDelay”  call 

Canceling  edges 

“intermpt”  call 

Edge  Delay  Times 

Time  argument  of  “waitDelay”  call 

Edge  Conditions 

“if  condition  block”,  wrapping  a 

“waitDelay”  or  “interrupt”  call. 

Edge  Arguments. 

“waitDelay”  or  “interrupt”  call 

arguments 

Table  I-l  Relationship  Between  EGs  and  Simkit 

2.  Human  Factors  Techniques  in  Software  Development 


a.  Usability 

Usability  is  defined  as  the  property  of  an  item  of  being  suitable  and 
eonvenient  to  use.  In  [SHN92],  Shneiderman  defines  five  eomponents  of  eomputer 
usability  as 

•  Ease  of  learning 

•  High  speed  of  user  task  performanee 

•  Eow  user  error  rate 

•  Subjeetive  user  satisfaetion 

•  User  retention  over  time 

Eor  the  users  of  an  interaetive  system,  “the  interfaee  is  the  applieation”. 
Eor  that  reason,  developing  good  interfaees  to  improve  user  aeeeptanee  of  the  produet  is 
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vital.  Clearly,  usability  does  not  mean  just  a  window  interface  since  many  windows 
applications  exhibit  a  very  low  level  of  usability. 

The  following  points  describe  some  techniques  that  can  be  used  to 
improve  usability  of  software  applications. 

b.  User-Centered  Development 

User-Centered  seeks  involvement  of  the  “final  user”  in  the  development 
process  from  the  first  stages  of  the  conception  of  the  system.  Many  systems  have  been 
developed  without  getting  inputs  from  the  “final  user”;  these  systems  usually  have  very 
poor  usability.  A  “final  user”  is  understood  to  be  the  one  who  eventually  utilizes  the 
application. 

Two  errors  are  possible  when  developing  applications  involving  humans. 
The  first  occurs  when  the  developer  is  actually  one  of  the  potential  users.  The  developer 
should  not  be  considered  a  “final  user”;  therefore,  the  application’s  usability  must  not  be 
assessed  by  the  developer,  unless  the  developer  is  the  only  potential  user. 

The  second  problem  appears  when  the  user  selected  to  assess  usability 
does  belong  to  the  organization  but  is  not  one  of  those  who  will  eventually  use  the  tool. 
As  an  illustration,  consider  an  application  that  deals  with  bank  accounts:  a  final  user 
could  be  a  bank  teller  but  not  a  bank  executive,  even  though  the  executive  could  be  the 
client  representative. 

c.  Prototyping 

Prototyping  is  the  best  solution  to  build  a  rapid  model  of  the  Graphical 
User  Interface  (GUI).  Prototypes  do  not  provide  functionality  but  show  how  this 
functionality  is  made  accessible.  Prototypes  are  usually  done  in  specialized  languages  or 
tools.  The  final  product  uses  the  lessons  learned  from  the  experiments. 

d.  Formative  and  Summative  Usability  Experiments 

Formative  usability  experiments  are  employed  to  get  inputs  from  the  user 
during  the  development.  These  experiments  are  performed  using  prototypes  of  the  entire 
application  or  other  prototypes  built  to  address  particular  issues. 
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Summative  experiments  are  performed  with  the  final  applieation,  or  an 
intermediate  deliverable  product.  The  intent  of  these  experiments  is  to  refine  the  final 
product  and  evaluate  its  usability  level. 

e.  Parallel  Development  (Application  vs.  GUI) 

The  separation  of  functionality  and  interface  development  provides  a  way 
to  obtain  a  more  workable  system.  Implementing  this  approach,  the  interface  decisions 
are  separated  from  the  functionality  decisions,  leading  to  a  more  decoupled  system.  A 
better  interface  can  be  achieved  since  it  is  not  as  tightly  constrained  by  functionality 
design  decisions. 

Figure  I-l  is  based  on  [HIX93,  p.  115].  This  graph  depicts  the  dual  process 
of  building  the  application  software  in  parallel  with  the  interface  software. 


Figure  I-l  Parallel  System  Development 


C.  OBJECTIVES  AND  RESEARCH  QUESTIONS 

The  whole  process  of  designing  and  translating  EGs  into  code  is  currently 
performed  manually.  This  procedure  produces  high  error  rates,  long  development  times, 
difficult  to  maintain  software,  and  user  dissatisfaction.  An  Integrated  Development 
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Environment  (IDE)  for  simulation  projects  can  automate  this  process.  An  IDE  tool  is  a 
suite  of  applications  whose  purpose  is  to  analyze,  design,  implement  and  debug 
simulation  models,  plan  and  execute  experiments,  and  finally,  to  study  the  results. 

An  important  component  of  an  IDE  for  simulation  projects  is  a  graphical  design 
tool.  The  Event  Graph  Graphical  Design  Tool  (EGGDT)  developed  here  allows  the  user 
to  construct  EGs  and  to  automatically  generate  code  skeletons  in  Java  using  Simkit.  The 
decision  to  build  the  EGGDT  was  undertaken  because  no  tool  for  designing  EGs  and 
generating  Java  code  using  Simkit  was  found.  The  objective  of  this  research  is  to  use 
Human  Eactors  techniques  and  guidelines  to  model  and  build  an  EGGDT  and  to  measure 
its  effectiveness. 

The  hypothesis  stated  here  is  that  the  use  of  these  kind  of  tools,  when  they  are 
properly  designed  by  means  of  user-centered  approaches,  improves  OR  analyst 
satisfaction  and  helps  to  enhance  OR  productivity.  To  test  this  hypothesis,  reactions  of 
the  potential  users  in  front  of  one  of  these  tools  were  observed  and  measured. 

D.  SCOPE  OF  THE  THESIS  AND  METHODOLOGY 

The  methodology  used  to  develop  the  EGGDT  was  based  on  the  following 
principles: 

•  Use  the  Unified  Modeling  Eanguage  (UME)  as  the  Analysis  and  Design  (A&D) 
graphical  modeling  language. 

•  Implement  an  iterative  process  based  on  Use-Cases  (UC). 

•  Plan  the  development  pursuing  software  usability. 

•  Design  the  application  as  a  component-based  model  found  by  utilizing  design 
patterns. 

1.  UML 

The  A&D  graphical  tool,  UMfi,  is  used  to  develop  software  systems.  This 
instrument  communicates  and  depicts  the  application’s  structure  helping  to  set 
architectural  alternatives  and  to  justify  decisions  of  A&D.  Einally,  UME  improves 


1  For  more  information  about  UML  see  references  [LAR98],  [FOWOO],  [RUM99] 
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maintainability  by  providing  a  means  to  generate  the  necessary  documentation. 
Document  generation  is  the  most  time-consuming  task  in  software  development. 

To  conduct  this  project  certain  parts  were  analyzed  and  designed  using  UML. 
Since,  in  this  project,  the  EGGDT  represents  the  work  of  only  one  developer, 
documentation  in  this  paper  is  not  comprehensive. 

The  UML  artifacts  used  to  perform  the  A&D  of  the  EGGDT  were 

a.  Analysis 

•  Use-Case  Diagrams  (UML-UC)  depicting  decompositions  of  the 
functionalities  required  in  the  system.  The  external  entities  playing  a 
role  in  the  system  can  also  be  included. 

•  Conceptual  Model  Diagrams  (UML-CM)  showing  the  high-level 
problem-domain  object-decomposition  of  the  system. 

•  Sequence  Diagrams  (UML-SD)  illustrating  the  high-level  operation 
calls  between  the  external  entities  and  the  system. 

b.  Design 

•  Class  Diagrams  (UML-CD)  explaining  the  design-level  class- 
decomposition  of  the  system. 

•  Collaboration  Diagrams  (UML-ColD)  describing  the  details  of  every 
system’s  method. 

•  State  Diagrams  (UML-StD)  showing  the  behavior  of  control  classes, 
such  as  mouse  controller  classes. 

2.  Iterative  Process 

The  purpose  of  adopting  a  software  development  process  is  to  establish  a 
workflow  that  specifies  a  series  of  steps  to  follow.  Eollowing  a  development  process 
guarantees  a  certain  discipline  and  order.  A  good  example  of  a  development  process  is 
the  iterative  process^;  this  is  a  good  choice  for  this  particular  project  (and  for  many 
others).  In  an  iterative  process,  the  overall  project  is  built  in  successive  iterations.  Erom 
each  iteration  a  product  is  released,  with  subsequent  stages  built  upon  the  ancestor 
products. 

During  this  research  only  one  iteration  of  the  EGGDT  was  performed.  A  Use- 
Case  approach  was  used  to  determine  what  to  develop  in  this  iteration.  The  more  critical 


^  For  more  information  about  software  development  process  see  for  example  [FOWOO],  [COK97]  or 
[JAB99]. 
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and  risky  UCs  were  chosen  to  ensure  that  the  core  functionality  of  the  EGGDT  and  those 
components  that  compromised  the  whole  feasibility  of  the  product  were  developed  first. 

3.  Software  Usability 

Usability  was  pursued  by  incorporating  several  techniques.  One  technique,  user- 
centered  development,  in  particular,  was  achieved  via  a  two-flow  development,  one  for 
the  application  software  and  another  for  the  interface  software.  Formative  experiments 
were  performed  to  assess  the  intermediate  products  and  compare  design  alternatives.  A 
summative  experiment  was  performed  to  determine  the  opinions  of  the  potential  users 
about  these  kinds  of  tools. 

4.  Component-Based  and  Pattern  Modeling 

The  use  of  patterns  is  essential  at  the  design  level.  Design  patterns  were 
introduced  by  [GAM95];  however,  the  version  used  during  this  research  is  in  [LAR98]. 
Patterns  provide  known  structures  and  lessons  learned,  that  is,  they  are  the  means  by 
which  developers  can  take  advantage  of  the  wisdom  and  experience  of  other  developers. 
During  the  software  design,  patterns  such  as  “Controller”,  “Expert”  or  “Creator”  were 
used. 


Figure  1-2  shows  the  UML  collaboration  diagram  (UML-ColD)  used  to  design  the 
functionality  of  creating  edges  with  the  EGGDT.  First,  consider  that  this  UML-CD  only 
deals  with  the  problem  of  creating  an  edge  at  the  functionality  level  not  at  the  interface 
level.  This  model  makes  no  reference  to  arcs  or  mouse  or  any  other  graphical  element. 

The  rectangles  with  a  bent  corner  such  as  those  used  in  Figure  1-2  refer  to  the 
patterns  that  assign  the  operations  to  the  different  classes  represented  by  shaded 
rectangles.  The  operations  names  are  written  over  the  lines  that  join  the  classes;  the 
arrows  indicate  the  direction  of  the  call.  For  example,  addOutgoingEdge()  is  a  method  of 
the  class  Event  called  from  a  object  of  the  class  GUIController.  This  method  has  been 
assigned  to  Event  because  Event  is  the  only  one  that  knows  about  its  own  outgoing  edges. 
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In  [BUSOO]  a  description  of  Component-based  modeling  is  found.  To  model  the 
EGGDT  this  technology  was  used  as  well  as  the  traditional  Object  Oriented  (00) 
approach.  In  a  component-based  system  the  architecture  is  based  in  interfaces  and  the  use 
of  the  listener-pattern  [BUSOO,  par.  5];  in  contrast  to  this,  in  classical  00  design  the 
architecture  is  based  on  inheritance. 

E.  CONCLUSION 

Simulation  is  a  major  tool  to  analyze  OR  problems.  Simulations  are  software 
applications  that  can  be  developed  with  the  help  of  CASE  tools.  The  use  of  these  tools  is 
intended  to  improve  user  satisfaction  when  creating  simulation  models.  However,  since 
no  simulation-modeling  tool  based  on  EGs  and  Simkit  was  commercially  available,  the 
EGGDT  was  developed. 

The  EGGDT  is  an  example  of  a  CASE  tool  that  allows  designing  simulations  with 
EGs  and  facilitates  their  implementation  in  Java  using  Simkit.  To  develop  the  EGGDT, 
human  factors  have  been  considered;  user-centered  techniques  and  formative  experiments 
have  been  used  to  ensure  usability. 
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Chapter  II  presents  an  example  of  a  DBS  model  of  a  system  depicted  with  EGs 
and  codified  in  Java  using  Simkit.  Chapter  III  details  the  EGGDT  requirements  and  the 
A&D  decisions.  The  next  two  Chapters,  IV  and  V,  describe  the  formative  experiments 
performed  to  ensure  usability  during  development  and  the  summative  experiment  carried 
out  to  evaluate  the  subjective  increase  in  user  satisfaction  using  these  kinds  of  tools. 
Chapter  VI  describes  the  summative  experiment  carried  out  to  test  whether  the 
simulation-design  tools  improve  OR  analyst  satisfaction  when  developing  simulations. 
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II.  EVENT  GRAPH  EXAMPLE:  RELIABILITY  SYSTEM 


A.  INTRODUCTION 

This  chapter  presents  an  Operations  Research  (OR)  problem  that  ean  be  modeled 
as  a  Diserete  Event  Simulation  (DES)  using  Event  Graphs  (EG).  This  example  shows  the 
utility  of  DES  models  and  EGs  to  solve  OR  problems.  The  EGs  in  this  chapter  were 
created  with  the  Event  Graph  Graphical  Design  Tool  (EGGDT)  Version  0.3.  The  skeleton 
of  the  Java  code  was  generated  by  the  EGGDT  from  the  EG  model. 

B.  DEFINITION  OF  THE  PROBLEM 


Consider  the  system  in  Figure  II- 1.  Computer  I  sends  information  to  Computer2 
across  a  network.  The  link  between  Computerl  and  Computer!  depends  on  three  nodes 
inside  the  network  (Nodes  1,  2  and  3)  that  have  specifie  failure  time  distributions.  These 
distributions  can  be  empirieal  but  are  known. 


F igure  II- 1  System  A 


Figure  II -2  shows  the  block  diagram  of  these  three  nodes  eonsidering  only  the 
eonneetions  that  affeet  the  link  between  Computerl  and  Computer!.  This  bloek  diagram 
approaeh  ean  be  used  to  study  reliability  issues.  For  this  example,  the  Measurement  Of 
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Performance  (MOP)  is  the  Mean  Time  To  Failure  (MTTF)  of  the  link  between 
Computerl  and  Computer2. 


Figure  II-2  Block  Diagram  of  System  A 


C.  VERSION  1 :  REPAIR  SERVICES  ARE  NOT  CONSIDERED 

A  DBS  model  of  System  A  is  shown  in  the  EG  of  Figure  II-3.  This  model  does 
not  consider  repair  services  of  the  nodes,  so  when  one  component  fails,  it  is  not  put  into 
service  again. 


Figure  II-3  EG  for  System  A  (Version  1) 
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The  EG  in  Figure  II-3  starts  with  the  “Run”  event  that  then  schedules  the  three 
“Nodes  Failure”  events.  The  “Node  1  Failure”  event  sets  the  state  variable  “Nl”  to 
“down”  to  indicate  that  the  node  is  not  working.  According  to  the  configuration  depicted 
in  the  block  diagram  of  Figure  II-2,  a  failure  in  “Node  1”  causes  the  link  to  fail; 
consequently,  “Node  1  Failure”  event  schedules  a  “Fink  Failure”  event  to  be  scheduled 
with  zero  delay  time. 

The  behaviors  of  the  events  “Node  2  Failure”  and  “Node  3  Failure”  are  similar. 
Each  sets  its  state  variable  to  “down”  and  schedules  a  “Fink  Failure”  event  if  the  other 
component  in  parallel  (N2  or  N3)  has  already  failed. 

To  calculate  the  MTTF,  a  number  of  replications  have  to  be  performed  recording 
the  time  the  “Fink  Failure”  event  occur  for  every  replication.  The  mean  of  these  times 
can  be  used  as  an  estimator  to  approximate  the  MTTF  of  the  link  if  the  number  of 
replications  is  large.  Similarly,  a  histogram  of  the  times  when  the  “Fink  Failure”  event 
occurred  for  every  replication  can  be  an  estimate  of  the  distribution  of  the  link  failures 
times. 

This  simple  problem  can  solved  using  the  mathematics  provided  by  the  reliability 
literature.  In  particular,  the  structure  function  of  System  A  is  given  in  Equation-II-1 

Equation-II-1  <1)(X)  =  min  [  Xi,  max  {  X2,  X3}  ] 

Xi,  X2  and  X3  are  binary  variables  that  equal  one  if  the  component  is  working  and 
zero  if  the  component  fails.  0(X)  has  two  possible  values  1  or  0  depending  on  whether 
the  link  is  up  or  not.  Based  on  Equation-II-1,  the  survival  function  (S(t))  for  this  system 
is  expressed  in  Equation-II-2. 

Equation-II-2  S(t)  =  Si(t)  [  1  -  { 1-  82(1) }  {1-  S3(t) )  ] 

Where  S(t)  is  the  probability  that  the  link  is  up  in  time  t;  the  terms  Si(t),  S2(t)  and 
S3(t)  are  the  particular  survival  functions  of  the  components.  To  get  the  MTTF  the 
integral  of  S(t)  from  0  to  infinity  is  used.  However,  the  integral  can  be  difficult  or 
impossible  to  evaluate  analytically,  so  numerical  methods  have  to  be  used.  In  addition, 
this  approach  does  not  consider  repair  services. 
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As  an  illustration  of  the  validity  of  the  simulation  approaeh,  let  consider  that  the 
survival  functions  Si(t),  S2(t)  and  83(0  present  exponential  distributions  with  mean  time 
between  failures  of  2.5  ,2.0  and  2.2  months.  Figure  II-4  shows  a  probability  plot  of  the 
exact  survival  function  S(t)  and  the  values  from  the  simulation.  The  simulation  was 
program  in  Java  using  Simkit  based  in  the  EG  of  Figure  II-3.  The  model  was  run  500 
times  to  get  the  plot  in  Figure  II-4.  The  departure  of  the  simulation  curve  from  the  exact 
curve  in  Figure  II-4  is  very  small. 

The  values  obtained  were 

Exact  MTTF=  1.54 

Simulation  MTTF  =  1.57 

95%  Cl  if  the  simulation  MTTF  =  [1.45,  1.69] 

The  exact  MTTF  is  included  in  the  95%  Cl  for  the  simulation  MTTF.  As  the 
number  of  iterations  increases,  the  simulation  MTTF  approaches  the  exact  MTTF.  For 
example  at  10,000  iterations,  the  simulation  MTTF  is  1.56. 


Figure  II-4  Plot  of  the  Exact  Survival  Function  vs.  the  Simulation  Output  Values  for 

System  A  (Version  1) 
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D.  VERSION  2:  REPAIR  SERVICES  ARE  CONSIDERED 

When  nodes  can  be  repaired  and  put  back  to  service,  the  mathematical  model  of 
the  previous  section  is  inadequate  to  capture  the  system  behavior.  However,  the 
simulation  model  can  be  easily  adjusted  to  accommodate  this  situation. 

The  assumptions  for  this  model  are: 

•  When  one  node  fails,  it  is  repaired  in  a  random  time  whose  probability 
distribution  is  known. 

•  The  nodes  continue  working  even  if  the  link  fails;  this  is  a  reasonable  assumption 
because  these  nodes  are  not  only  serving  this  link  so  they  are  kept  working  even  if 
the  link  is  broken. 

•  Repair  facilities  exist  to  serve  every  node  independently,  i.e.,  repair  queues  do  not 
exist.  The  distributions  of  the  repair  times  are  known. 


I  L 


— clown  AND  (Nl  —  up  AND  |  L  up  AND  (Nl  clown  OR 

(N2==upOR  N3==up))  I  (N2==clown  AND  N3=^clown))  | 


Figure  II-5  EG  for  System  A  (Version  2:Repair  services  considered) 
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The  EG  of  Figure  II-5  eommenees  when  the  “Run”  event  sehedules  all  the  “Start 
Working”  events.  These  events  set  the  state  variables  “Nl”,  “N2”  and  “N3”  to  “up”  to 
indicate  that  the  nodes  are  working.  When  the  necessary  nodes  are  up(i.e.,  “Node  1”  and 
“Node  2”  or  “Node  3”),  the  “Connection  Resumed”  event  is  fired;  this  event  sets  the  state 
variable  L,  which  represents  the  state  of  the  link,  to  “up”. 

Each  “Start  Working”  event  also  schedules  its  own  “Node  Failure”  event  with 
delays  determined  by  the  corresponding  “Time  to  Fail”  random  variable.  “Node  Failure” 
events  schedule  their  own  “Start  Working”  events  using  their  “Time  to  Repair”  random 
variables.  “Connection  Failure”  is  scheduled  by  one  of  the  “Node  Failure”  events  when 
the  link  is  up  and  “Node  1”  is  down  or  “Node  2”  and  “Node  3”  are  both  down. 

This  cycle  continues  until  the  simulation  is  stopped  based  on  time  or  number  of 
failures.  These  details  depend  on  the  design  of  the  experiments  to  be  performed.  To 
obtain  the  MTTF  the  same  method  explained  for  Version  1  can  be  used.  For  this 
particular  problem,  the  analytical  solution  is  difficult  or  impossible  because  of  the 
distributions  of  the  times  to  failure  and  repair. 

E.  EVENT-LIST 

To  illustrate  how  the  event-list  works  using  the  DES  model  of  Version  2  Figure 
II-5,  suppose  the  “Node  1”  and  “Node  3”  are  currently  working  and  “Node  2”  is  being 
repaired.  A  possible  content  for  the  event-list  is  detailed  in  Table  II-l. 


Time 

Event 

Parameters 

10 

Node  1  Failure 

i  =  1 

13 

Node  3  Failure 

i  =  3 

16 

Start  Working  2 

i  =  2 

Table  II- 1  Example  of  a  Possible  Event-Fist  for  System  A 

Table  II-2  shows  the  event-list  in  time  10  after  being  modified  by  execution  of  the 
“Node  1  Failure”  event. 
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Time 

Event 

Parameters 

10 

Connection  Failure 

13 

Node  3  Failure 

i  =  3 

14 

Start  Working  1 

i=  1 

16 

Start  Working  2 

i  =  2 

Table  II-2  Example  of  a  Possible  Event-List  for  System  A  (After  time  10) 

F.  JAVA  SOURCE  CODE 

Eor  the  sake  of  elarity  of  the  example,  the  EG  in  Eigure  II-5  was  depicted  in  a 
high  level  manner.  The  Java  code  described  below  is  more  detailed  but  is  based  on  this 
EG.  The  Java  source  code  from  this  model  is  encapsulated  in  a  class  that  extends  the 
Simulation  Kit  (Simkit)  simulation  entity  abstract  class. 

import  simkit.*; 

public  class  SimpleReliabilityModel02  extends  SimEntityBase  { 

The  main  code  of  this  class  is  explained  below.  The  state  variables  and  simulation 
parameters  are  declared  as  private  instance  variables  of  the  class 

//  state  Variables 

private  boolean  L  =  false  ;  //  true  =  up  &  false  =  down 
private  Boolean []  N  =  new  Boolean [3];  //  set  all  to  false 

//  Simulation  Variables 

private  randomVariate [ ]  timeToFail  =  new  randomVariate [ 3 ] ; 

Eor  every  event,  a  “do”  method  is  implemented. 

/  *  * 

*/ 

public  void  doStartWorking ( int  1)  { 

N[i]=  true; 

if (L==  false  &&  (Nl==true  && (N2==true  | |  N3==true) )  )  { 

waitDelay ( "ConnectionResumed" ,  0.0) ; 

} 

waitDelay ( "NodeFailure"  ,  TimeToFail [ 1 ]. generate () ,  i) ; 

} 

/  *  * 

*/ 

public  void  doNodeFailure  (int  i)  { 

N[i]=  false; 

waitDelay ( "Startworking"  ,  TimeToRepair [ i ]. generate (), i  ); 
if  (  L==  true  &&  (Nl==false  | |  (N2==false  &&  N3==false) ) )  { 

waitDelay ( "ConnectionFailure"  ,  0.0  ); 

} 
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} 

!  -k-k 
*/ 

public  void  doConnectionResumed  (  )  { 

L=true; 

} 

!  kk 
k  / 

public  void  doConnectionFailure  (  )  { 

L=false; 

//  Insert  the  necessary  code  to  record  the  time  the  link 
//  has  been  up. 

} 

The  “waitDelay”  calls  implement  the  outgoing  edges  of  the  event.  The  “if 
statements”  are  a  consequence  of  the  edge  conditions. 

G.  CONCLUSION 

This  example  shows  that  systems  can  be  modeled  using  the  DBS  paradigm.  DBS 
models  can  be  represented  as  BGs  and  from  them  executable  simulations  can  be  obtained. 
The  analytical  solutions  for  a  wide  range  of  problems  are  not  available.  Bor  example, 
systems  whose  behavior  involves  stochastic  processes  are  very  difficult  or  impossible  to 
abstract  as  mathematical  models;  simulation  is  the  only  resource  in  these  kinds  of 
situations.  DBS  simulations  are  also  easily  expandable  and  accept  a  broad  diversity  of 
input  parameters.  In  other  words,  DBS  simulations  are  a  flexible  tool  to  study  the 
behavior  of  systems.  The  next  chapter  details  the  requirements  of  the  BGGDT. 
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III.  EGGDT  SOFTWARE  REQUIREMENTS,  ANALYSIS  AND 

DESIGN 

A.  INTRODUCTION 

The  purpose  of  this  ehapter  is  to  detail  the  requirements  for  the  Event  Graph 
Graphical  Design  Tool  (EGGDT)  and  the  more  important  Analysis  and  Design  (A&D) 
decisions  .  The  goal  of  the  EGGDT  is  to  allow  depiction  of  Event  Graphs  (EG).  Eor  more 
information  about  EGs  see  [BUSOl],  [BUS96],  [SCH83]  or  [SCH95]. 

B,  EGGDT  REQUIREMENTS 

1,  Overview  Statement 

The  focus  of  this  particular  research  is  on  the  analysis,  design  and  implementation 
of  a  computer  tool  aimed  at  supporting  simulation  design  and  implementation  using 
Event  Graphs  (EGs),  Java  and  the  Simulation  Kit  (Simkit). 

The  intent  of  the  EGGDT  tool  is  to  help  Operations  Research  (OR)  analysts  in  the 
design,  implementation,  and  debugging  of  simulation  software.  The  system  is  envisioned 
as  a  graphical  tool  to  draw  EGs  for  eventual  translation  into  code  thereby  reducing  errors 
and  obtaining  documentation  from  the  models. 

An  EG  and  a  Simkit  application  have  a  straightforward  correspondence.  Eor  every 
element  in  an  EG  specific  Java  code  exists  in  the  Simkit  program.  This  correspondence 
between  an  EG  and  Simkit  code  is  shown  in  Table  I-l.  See  Appendix  A  for  details  and  a 
simple  example. 

The  functional  and  interface  requirements  for  the  EGGDT  have  been  considered 
separately.  Separation  between  Graphical  User  Interface  (GUI)  and  functionality  allows 
for  a  user-centered  development  approach.  As  discussed  in  the  introductory  chapter,  the 
user-centered  method  is  essential  to  meet  user  expectations. 


3  The  study  contained  in  this  chapter  started  in  two  group  projects  for  the  IS-3020  and  MV -4202 
courses.  The  rest  of  participants  for  the  IS-3020  project  were:  LtJG  Gokhan  Ozkan,  Maj  Mark  Harrington, 
Maj  Raj  Mohan.  LCdr  Paulo  Silva  was  the  other  participant  in  the  MV -4202  project. 


19 


Initially,  the  EGGDT  is  aimed  at  OR  students  and  educators  of  the  Naval 
Postgraduate  School  (NPS).  Eventually,  other  users  will  be  OR  analysts  or  those 
involved  with  modeling  and  simulation  studies  and  practices.  The  users  of  the  EGGDT 
are  assumed  to  have  a  background  in  and  be  familiar  with  OR  methods  and  simulation 
design,  particularly  EG  modeling. 

2.  Goals 

The  goal  of  the  EGGDT  project  is  to: 

•  Provide  a  friendly  GUI  to  develop  EGs. 

•  Automate  the  process  of  generating  simulation  code  from  a  known  EG. 

•  Provide  a  tool  to  document  the  simulation  details. 

3.  Implementation 

The  EGGDT  was  implemented  in  Java  using  the  Swing  and  Java  2D  (J2D) 
libraries  because  Java  is  a  modern,  platform- independent  object  oriented  programming 
language.  Additionally,  the  Swing  and  J2D  libraries  offer  a  wide  variety  of  functionalities 
for  GUIs.  As  an  extra  benefit,  Java  is  the  Internet  programming  language;  therefore, 
even  though  the  first  version  of  the  EGGDT  is  a  stand-alone  computer  application, 
implementing  it  as  an  Internet  application  is  a  straightforward  extension. 

Another  reason  to  adopt  Java  is  that  Simkit  is  written  in  Java,  so  the 
communication  between  the  tool  and  Simkit  is  smooth.  Java  virtual  machines  are 
available  for  all  major  platforms,  including  MS-Windows,  Einux,  Eree  BSD,  Mac  OS  X 
and  commercial  Unix.  Java  virtual  machines  are  also  available  in  handheld  devices,  such 
as  Palm  Pilot.  It  is  projected  that  by  2002  there  will  be  more  Java  than  C++  developers. 

4.  Functionality  Requirements 

Eunctionality  requirements  refer  to  the  desired  internal  behavior  of  the  system 
with  no  reference  to  the  way  these  functionalities  are  provided  to  the  user.  The  external 
behavior  is  detailed  in  the  interface  requirements. 

The  functionality  requirements  are  broken  down  in  Appendix  B. 

The  most  important  functionality  requisites  are: 
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•  Allow  the  user  to  ereate,  delete  and  modify  EGs  and  all  its  elements 

(events,  edges  and  simulation  variables). 

•  Generate  the  skeleton  of  Java  class  from  the  EG. 

5.  GUI  Requirements 

GUI  requirements  refer  only  to  the  external  appearance  and  behavior  of  the 
EGGDT.  Clarifying  the  distinction  between  functionality  and  interface  entails  realizing 
that  even  though  an  EG  contains  events,  edges  and  simulation  variables,  its  representation 
can  take  may  forms.  Eor  example,  an  EG  can  be  represented  as  a  picture  with  circles  and 
arcs,  a  Java  class,  a  list  of  the  elements  in  an  XML  (Extensible  Markup  Language)  file,  a 
text  file,  an  HTML  file  or  in  any  other  form  storing  the  information  contained  in  the  EG. 

The  EGGDT  requirements  of  the  GUI  specify  that  the  approach  chosen  represents 
EGs  as  pictures  containing  circles  and  arcs  for  the  events  and  edges,  and  some  kind  of 
tabular  structure  for  the  simulation  variables. 

Appendix  B  specifies  the  interface  functionalities  that  the  EGGDT  has  to  provide. 

C.  EGGDT  ANALYSIS 

During  this  research  only  one  iteration  of  the  EGGDT  project  was  performed.  A 
Use-Case  (UC)  approach  was  followed  to  select  those  functionalities  that  should  be 
implemented;  the  most  essential  or  risky  UCs  were  selected. 

1.  Functionality  Analysis 

The  functionality  analysis  describes  how  the  basic  functions  of  the  EGGDT 
have  been  broken  down;  no  reference  to  graphical  elements  is  made  in  this  kind  of  study. 

a.  Use-Cases 

As  discussed  in  Chapter  I,  UML  UCs  have  been  used  to  document  the 
analysis  of  the  EGGDT.  The  basic  functionality  of  the  EGGDT  is  covered  in  the  UC 
diagram  (UML-UCD)  of  Eigure  III-l. 

The  following  are  the  UCs  contained  in  this  diagram: 

•  “Create  EG”  (see  Table  III-l). 

•  “Create  EG  element”.  EG  elements  are  events,  edges  and  simulation 
variables  (see  Table  III-2). 
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•  “Modify  EG  elements”  content. 

•  “Delete  EG  Element”. 

•  “Generate  Java  code”  using  the  correspondence  articulated  Table  I-l . 


Generate  Java 


Eigure  III-l  EIME-EICD.  Basic  EGGDT  Eunctionality 

These  UCs  were  considered  to  cover  the  core  requirements  of  the 
EGGDT.  The  functionality  requirements  were  enumerated  in  paragraph  II-B-0. 

Once  UCs  were  identified  they  were  detailed  in  the  extended  UCs.  As  an 
illustration  to  explain  UCs,  the  extended  UCs  for  “Create  EG”  and  “Create  EG  Element” 
are  presented  in  Table  III-l  and  Table  III-2. 
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Use  Case: 

Create  EG. 

Actors: 

Analyst 

Purpose: 

Create  a  new  EG.  Includes  the  EG  properties  like  name,  project,  package  and 

so  on. 

Typical  Course  of  Events 

Actor  Action 

System  Response 

1.  The  Analyst  initiates  an  EG  entering  all  2.  Creates  a  new  EG. 

necessary  data. 

Alternative  Courses. 

Eine  2.  If  the  name  of  the  EG  is  not  unique  in  the  package,  the  system  prompts  an  error  msg. 

Table  III-l  Extended  UC  Create  EG 

Use  Case: 

Create  EG  element 

Actors: 

Analyst 

Purpose: 

Create  a  new  EG  element.  Allows  the  user  to  attach  element’s  properties. 
The  system  must  update  the  list  of  elements 

Typical  Course  of  Events 

Actor  Action 

System  Response 

1.  The  Analyst  initiates  the  creation  of  an 

event,  edge  or  simulation  variable 

Case  “Create  Event” 

2.  Creates  the  new  event  instance. 

3.  Updates  the  list  of  events. 

Case  “Create  Edge” 

2.  The  Analyst  specifies  the  source  and  target  3.  Creates  the  new  edge  instance. 

event  of  the  edge.  4  Updates  the  outgoing  edges  list  of  the  source 

event. 

5  Includes  a  reference  of  the  target  event  in  the 
edge. 

6.  Updates  the  list  of  edges. 

Case  “Create  Simulation  Variable” 

2.  The  Analyst  specifies  if  it  is  a  State  Variable  3.  Creates  the  new  simulation  variable  instance, 
or  Simulation  Parameter.  6,  Updates  the  list  of  simulation  variables. 

Alternative  Courses. 

Line  2.  If  the  name  of  the  event  or  the  simulation  variable  is  not  unique  in  the  EG,  the  systems 

prompts  an  error  msg. _ 

Table  III-2  Extended  UC  Create  EG  Element 
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b.  Conceptual  Model 

The  Conceptual  Model  (UML-CM)  in  Figure  III-2  summarizes  and 
identifies  the  primary  classes  related  with  core  functionality  and  their  associations.  The 
EG  is  composed  of  events,  edges  and  simulation  variables.  Events  have  outgoing  edges 
which  point  to  target  events.  Simulation  variables  can  be  state  variables  or  simulation 
parameters.  Events  actions  modify  state  variables  and  edges  conditions  consult  their 
value. 

2,  Task  Analysis 

a.  Use-Cases 

Task  analysis  addresses  the  problem  of  defining  the  external  appearance 
and  behavior  of  the  EGGDT.  Eigure  III-3  shows  the  UML-UCD  corresponding  with  the 
interface  functionality  required. 


Eigure  III-2  EiME-CM  Diagram.  Basic  EGGDT  Model 
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The  UCs  identified  are: 


•  “Initiate  the  EGGDT”  (see  Table  III-3). 

•  “Manipulate  Graphical  Element”  includes  the  creation  and 
modification  of  these  elements.  Graphical  elements  are  circles  and  arcs 
representing  the  events  and  edges  respectively. 

•  “Manipulate  simulation  variables”  that  are  included  in  the  EGGDT  for 
appropriate  representation,  for  example,  a  table  or  tree. 

•  “Manipulate  EG  files”  implementing  the  “Open”,  “Save”,  “Rename” 
and  “Saved  As”  services. 


As  an  example,  the  extended  UC  for  Initiate  EGGDT  is  provided. 
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Use  Case:  Initiate  EGGDT. 

Actors:  Analyst,  Visual  Output  System 

Purpose:  Launch  the  Graphical  Interface  Tool.  Initialize  all  the  variables.  Load  the 

options  for  the  session  and  display  the  GUI  with  a  start  up  window. 

Typical  Course  of  Events 

Actor  Action  System  Response 

1.  The  Analyst  starts  the  EGGDT  program.  2.  Initiates  the  graphical  tool. 

3.  Prompts  the  user  to  create  a  New  EG  or 
Open  an  existing  EG. 

4.  The  Analyst  enters  the  selected  option. 

Case  “Create  New  EG” 

5.  Displays  the  EG  properties  window. 

6.  The  Analyst  modifies  the  properties  of  the  7.  Opens  a  new  blank  EG  in  the  Visual  Output 
EG.  System. 

Case  “Open  EG” 

5.  Displays  an  open  file  window. 

6.  The  Analyst  selects  the  EG  file  to  open.  7.  Loads  the  selected  EG  elements  and  shows 

them  in  the  Visual  Output  System. 

Alternative  Courses. 

Line  6.  The  system  prompts  an  error  message,  if  any  EG  property  is  unacceptable  or  the  selected 

EG  file  to  open  is  not  valid. _ 

Table  III-3  Extended  UC  Initiate  EGGDT 

b.  Conceptual  Model 

The  UML-CM  in  Eigure  III-4  summarizes  and  identifies  the  primary 
elasses  related  with  the  interfaee.  The  GUI  eontroller  manipulates  two  different  interfaee 
areas.  The  “Drawing  Area”  displays  the  representations  of  edges  and  event-  ares  and 
eireles.  The  “Area  of  Variables”  provides  a  means  to  manipulate  simulation  parameters 
and  state  variables. 
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Figure  III-4  UML-CM  Diagram.  EGGDT  Interface  Model 

D,  EGGDT  DESIGN 

The  design  phase  pursues  the  building  of  the  software  architecture.  Many  tools 
are  available  to  help  perform  this  task;  Collaboration  Diagrams  (UML-ColD)  and  State 
Machine  Diagrams  (UML-SM)  were  used  in  the  development  of  the  EGGDT.  UME- 
ColDs  allow  easy  access  in  applying  patterns,  as  discussed  in  Chapter  I. 

Following  are  some  examples  showing  the  approach  used  to  design  the  EGGDT. 
To  demonstrate  the  different  approaches  used  to  design  the  application  and  interface,  two 
parallel  methods,  one  from  each  domain,  have  been  chosen. 

1,  System  Method  “Create  Edge” 

“Create  Edge”  is  a  method  in  the  domain  of  the  application;  no  interface 
considerations  are  contemplated.  Eigure  III-5  shows  the  UME-ColD  of  this  method. 
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1:  CreateEdge(source,  target) 


2:  create(target) 


Figure  III-5  UML-ColD.  Create  Edge 


The  GUI  initiates  a  request  to  create  an  edge.  The  event  graph  object  receives  the 
source  and  target  event  for  the  new  edge.  It  creates  an  edge  passing  a  reference  of  the 
target  event.  Finally,  the  event  graph  object  passes  the  new  edge  reference  to  be  included 
in  the  list  of  edges  and  in  the  list  of  outgoing  edges  of  the  source  event. 

2,  System  Method  “Create  Edge  Figure” 

The  method  “Create  Edge  Figure”  is  part  of  the  interface  domain;  therefore,  the 
output  and  input  system  are  considered.  Figure  III-6  contains  the  UMF-ColD  for  this 
method.  This  method  is  called  by  the  Analyst,  who  passes  the  source  and  target  events 
and  the  path  of  the  arc.  The  GUI  controller  object  creates  an  edge  figure  instance 
providing  the  target  event  and  path.  The  edge  is  included  in  the  list  of  edge  figures  and  in 
the  list  of  outgoing  edges  of  the  source  event.  Finally,  the  output  system  is  requested  to 
repaint. 
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Figure  III-6  UML-ColD.  Create  Edge  Figure 


3,  Mouse  Control  Behavior 

State  machines  were  found  to  be  very  useful  in  determining  and  depicting  the 
behavior  of  control  elements.  UML  State  Machines  Diagrams  (UML-SMD)  were  used. 

The  mouse  control  behavior  is  depicted  in  the  UML-SMD  of  Figure  III-7.  The 
states  of  the  mouse  controller  are  represented  by  rounded  rectangles.  The  transitions 
between  states  are  represented  by  arcs.  The  events  that  trigger  these  transitions  are 
expressed  as  labels  in  the  arcs.  These  events  are: 

•  mD  .-  dragging  the  left  button  on  the  mouse 

•  mP.-  pressing  the  left  button  on  the  mouse 

•  mR.-  releasing  a  button  on  the  mouse 

•  mDC. -double  clicking  the  left  button  on  the  mouse 

•  mCD.-  dragging  the  central  button  on  the  mouse 

•  mM.-  moving  the  mouse 

As  an  example  of  mouse  control  modeling,  consider  the  behavior  of  the  controller 
when  creating  an  edge.  The  controller  starts  in  an  “idle”  state;  when  the  mouse’s  central 
button  is  dragged  over  an  event  or  the  mouse’s  left  button  is  dragged  over  an  event  while 
pressing  the  keyboard’s  control-key,  the  system  transits  to  the  “creating  edge”  state. 
Each  time  a  button  on  the  mouse  is  released  over  an  empty  area,  a  new  inflection  point  is 
created,  but  the  controller  does  not  change  its  state. 
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Figure  III-7  UML-SMD.  Mouse  Controller 

If  the  mouse  is  released  over  an  event,  an  edge  is  ereated  and  the  eontroller 
transits  to  “seleeted  edge”  state.  Alternatively,  if  the  mouse  is  double-clieked  over  an 
empty  area,  the  operation  will  be  eancelled  and  the  controller  transits  to  the  “selected 
event”  state. 

E,  CONCLUSION 

This  chapter  provided  an  overview  of  the  development  of  the  first  iteration  of  the 
EGGDT  project.  The  requirements  were  broken  down  into  application  and  interface. 
During  the  analysis  phase,  a  UC  approach  was  used  to  determine  which  requirements  to 
develop  in  this  iteration.  Extended  UCs  describe  the  details  of  every  functionality 
required.  UME-CMs  identify  the  primary  objects  in  the  application  and  interface  domain. 

Einally,  for  the  design  phase  UME-ColDs  and  UML-SMDs  were  used.  UML- 
ColDs  provided  a  means  of  getting  inside  the  details  of  every  system  method  and  an  easy 
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way  to  implement  patterns.  UML-SMDs  were  found  benefieial  in  representing  the 
behavior  of  control  classes,  such  as  the  mouse  controller. 

By  applying  these  techniques,  separating  application  and  interface  domains  at  all 
levels  of  development  and  using  UML  artifacts  to  depict  the  A&D  decisions,  a  more 
decoupled  and  maintainable  software  architecture  has  been  achieved.  The  next  chapter 
explains  how  usability  was  pursued  through  formative  experiments  during  the 
development. 
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IV.  FIRST  EXPERIMENT:  USER  RESPONSE  TO  THE 
PROPOSED  TOOL  GRAPHICAL  USER  INTERFACE  (GUI) 

DESIGN 


A,  INTRODUCTION 

Two  formative  experiments  were  eondueted  during  the  design  of  the  Event  Graph 
Graphieal  Design  Tool  (EGGDT)  to  ensure  that  the  final  produet  would  fulfill  user 
expeetations. 

The  goal  of  the  first  experiment^  was  to  evaluate  the  quality  of  the  initial  design 
of  the  EGGDT.  The  rest  of  this  ehapter  deseribes  this  experiment  discussing  its  influence 
on  the  EGGDT  final  design.  The  second  experiment  is  covered  in  Chapter  V. 

B,  PURPOSE  OF  THE  EXPERIMENT 

This  experiment  did  not  attempt  to  be  a  comprehensive  test  of  the  tool,  but  a 
device  to  obtain  feedback  from  the  potential  users  about  the  approach  chosen  for  the  GUI 
of  the  EGGDT.  Additionally,  the  experiment  provided  an  initial  opportunity  to  use  the 
techniques  of  Human  Eactors  necessary  to  accomplish  usability.  These  techniques 
include  design  of  experiments,  task  analysis,  design  of  experimental  protocol, 
development  of  prototypes  and  analysis  of  the  data  from  the  experiments. 

When  this  experiment  was  conducted,  the  GUI  was  still  in  the  initial  phase, 
therefore  formative  evaluation  was  used.  Since  this  was  the  first  usability  experiment,  this 
evaluation  was  expected  to  introduce  important  design  changes.  The  following  sections 
of  this  chapter  will  summarize  the  evaluation  work. 

C,  DESIGN  OF  THE  EXPERIMENT 

Usability  goals  were  set  prior  to  appraising  the  GUI.  These  goals  served  as  a 
reference  point  for  the  evaluation  of  the  GUI.  In  addition,  they  acted  as  a  tangible 
measurement  of  the  usability  success  level  for  the  interface  design. 


^  This  experiment  was  part  of  the  final  project  for  MV-4203  (Prof  Rudy  Darken)  in  team  with  LCdr. 
Paulo  Silva  (Portuguese  Navy). 
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Hix  suggests  in  [HIX93]  that  first  time  developers  should  not  start  making  a  large 
usability  specification  table.  Therefore,  only  the  following  three  usability  attributes  for 
assessment  were  included;  “initial  performance”,  “learnability”  and  “first  impression”. 
The  complete  Usability  Specification  Table  is  presented  in  Appendix  C. 

1,  Participants 

The  word  “participant”  is  used  to  refer  to  the  subjects  who  collaborated  in  the 
evaluation  experiment.  Five  participants  involved  in  five  evaluation  sessions  were 
representative  of  the  users  group  for  the  EGGDT  as  defined  in  Chapter  III. 

2,  Tasks  to  Perform 

A  set  of  tasks  was  selected  for  the  purpose  of  the  interface  evaluation.  These  tasks 
are  a  subset  of  those  identified  in  the  task  analysis  phase  (Appendix  B),  and  represent 
typical  tasks  users  performance.  Appendix  C  lists  all  tasks  that  the  participants  were 
required  to  perform. 

Most  tasks  were  benchmark  tasks,  which  means  they  were  used  to  obtain  some 
quantitative  measurement  of  usability  performance  of  a  given  interface  attribute.  Other 
tasks  were  not  quantitative  but  were  included  to  complement  the  sequence  of  events  in 
the  experiment. 

D.  DESIGN  OF  THE  PROTOTYPE 

The  prototype  presented  the  following  capabilities: 

•  The  prototype  showed  full  button  and  menu  bars;  however,  only  the  following 
buttons  were  enabled; 


• 

“Select  toof  ’ 

• 

“Create  Event” 

• 

“Create  Edge” 

0  o| 

•  The  intent  of  the  prototype  was  to  test  the  GUI  approach  even  though  the 
necessary  functionalities  were  not  yet  implemented. 
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Figure  IV- 1  Snapshot  of  the  EGGDT  Prototype 


•  Modes; 

•  Events  eould  be  ereated  by  seleeting  “Create  Event”  and  elieking  in  the 
drawing  area. 

•  Edges  could  be  drawn  by  selecting  “Create  Edge”  and  clicking  first  in  the 
source  event  and  then  in  the  target  event.  Middle  points  could  not  be 
defined.  Three  edge  interior  points  were  provided  for  reshaping,  but  they 
could  not  be  deleted. 

•  The  “select  toof  ’  had  to  be  activated  to  move  events  and  reshape  edges. 

•  The  prototype  used  the  permanent  modality  method,  i.e.,  when  the  “Create 
Event”  button  was  pressed,  it  stayed  pressed,  so  multiple  events  could  be 
created.  In  a  temporary  modality  method,  the  button  would  have  to  be 
pressed  to  create  each  event. 

•  The  status  bar  offered  help  depending  on  the  active  mode. 

No  further  functionalities  were  included  in  this  first  prototype.  The  prototype’s 
layout  is  shown  in  Eigure  IV- 1. 
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E, 


EXPERIMENTAL  PROTOCOL 


The  standard  protocol  for  performing  evaluation  with  humans  requires  a  consent 
form  to  be  signed  by  the  participant.  Appendix  C  shows  the  consent  that  was  conveyed  to 
each  participant  to  read  and  sign. 

A  sheet  with  introductory  instructional  remarks  was  prepared,  given  to  each 
participant,  and  read  aloud  by  the  evaluator.  Any  additional  verbal  explanation  about  the 
interface  was  given  in  such  a  way  that  all  participants  obtained  the  same  information, 
thus  ensuring  consistency  among  participants.  Appendix  C  includes  a  sample  of  the 
instructions. 

1,  Pilot  Testing 

When  all  pieces  of  the  setup  were  together,  a  pilot  test  was  performed  clarifying 
the  challenge  of  a  realistic  observation  of  errors.  Since  the  experiment’s  environment 
setup  was  simplistic,  without  videotaping,  audiotaping  or  interface  interaction  recording 
capability  embodied  in  the  prototype,  the  evaluation  was  simplified.  The  performance 
data  focused  primarily  on  the  required  time  to  complete  tasks.  Nevertheless,  the  evaluator 
also  recorded  qualitative  information  about  the  types  of  errors  observed. 

Once  the  necessary  modifications  on  the  task  list  and  data  collection  forms  were 
introduced,  a  second  pilot  test  was  performed.  A  new  problem  was  observed  when  users 
responded  to  the  questionnaire.  Hix  stated  in  [HIX93]  that  participants  should  clearly 
understand  that  the  evaluation  is  not  proposed  to  assess  them,  therefore  they  should  not 
fear  any  consequences  of  a  “bad”  or  “good”  performance.  In  this  academic  environment, 
where  all  selected  participants  were  students  (sometimes  classmates  of  the  principal 
investigator),  participants  may  have  been  tempted  to  provide  higher  scores  on  the 
questionnaire.  As  a  result,  the  following  message  was  introduced  at  the  beginning  of  the 
questionnaire:  REMEMBER,  YOU  ARE  NOT  EVALUATING  US  BUT  THE  TOOL. 
SINCERITY  IS  APPRECIATED. 

2.  Evaluation  Sessions 

Having  completed  two  pilot  sessions  and  introduced  some  modifications  to  the 

required  material,  the  evaluation  sessions  were  performed.  Live  participants  took  part  in 
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separate  sessions  eondueted  in  the  OR  simulation  laboratory  in  Glasgow  Hall  at  the 
Naval  Postgraduate  Sehool  (NPS).  Participants  were  first  briefed  about  the  purpose  of  the 
experiment,  and  then  given  a  copy  of  the  Session  Instructions  (Appendix  C),  which  the 
experiment’s  controller  read  aloud.  Participants  also  read  and  signed  the  consent  sheet 
(Appendix  C). 

F.  STATISTICAL  ANALYSIS 

The  following  table  summarizes  quantitative  data  extracted  from  the  Data 
Collection  Forms.  For  each  of  these  tasks  a  statistical  analysis  is  presented,  including 
mean,  standard  deviation  and  95%  confidence  interval  (Cl). 


Benchmark  Tasks 

Participant 

1 

2 

3 

4 

5 

6 

P1 

80.0 

113.0 

60.0 

10.0 

14.0 

37.0 

P2 

57.0 

120.0 

20.0 

9.0 

20.0 

21.0 

P3 

49.0 

120.0 

99.0 

2.0 

22.0 

37.0 

P4 

43.0 

47.0 

56.0 

4.0 

11.0 

28.0 

P5 

47.0 

17.  0 

50.0 

45.0 

14.0 

33.0 

Mean 

55.2 

83.4 

57.  0 

14.0 

16.2 

31.2 

Stand  Deviation 

14.8 

48.2 

28.3 

7.7 

4.6 

6.8 

95%  a 

36.9 

23.6 

21.9 

0.0 

10.5 

22.8 

73.5 

143.2 

92.1 

35.9 

21.9 

39.6 

Table  IV-1  Benchmark  Times  in  Seconds 


In  Table  IV-1  : 

•  The  participants’  responses  are  assumed  to  be  independent  and  identically 
distributed  (iid). 

•  CIs  are  calculated  using  t-student  distribution  with  4  degrees  of  freedom. 

•  This  small  sample  results  in  large  variances  and  wide  confidence  intervals. 
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Figure  IV-2  shows  a  box-plot  of  the  benchmark  tasks’  times.  Every  participant 
performed  similarly  in  all  tasks  except  for  task  2  showing  a  great  variance  (83.4  sec). 
This  task  was  “Create  an  Edge  between  events  ‘Run’  and  ‘Arrival’  with  delay  time  ‘ta’  ”. 
As  discussed  later,  the  edge  creation  method  presented  difficulties;  it  was  not  intuitive,  so 
some  participants  were  able  to  create  edges  very  fast,  whereas  others  required  help  to 
complete  the  task. 


Figure  IV-2  Box-plot  of  the  Times  for  Benchmark  Task 


1,  Usability  Attribute  “Learnability” 

Eeamability  was  measured  through  the  comparison  between  benchmark  tasks  2 
and  6  (create  an  edge).  Both  were  very  similar  tasks,  one  performed  at  the  beginning  of 
the  experiment  and  the  other  at  the  end. 

To  test  the  null  hypothesis  that  both  means  are  the  same,  a  paired  t-test  was 
performed.  This  resulted  in  a  p-value  of  0.0743  and  a  95%  Cl  of  [-8.18,  112.58].  Even 
though  the  Cl  includes  0  for  a  0.5  level  of  confidence,  the  difference  between  means  is 
considered  significant. 

In  conclusion,  participants  in  the  experiment  generally  learned  the  tool  easily  (the 
experiment  only  lasted  20  minutes)  and  performed  tasks  faster  at  the  end  of  the  session. 
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Figure  IV-3  illustrates  the  difference  in  performing  both  similar  tasks.  With  one 
exception,  all  participants  performed  better  in  the  second  task. 


Comparasion  of  Bench mai1(s  02  and  06 


Participant 


Figure  IV-3  Differences  in  Performing  Both  Tasks  of  Creating  Edges 

2,  Usability  Attribute  “First  Impression” 

The  “First  Impression”  usability  attribute  measurement  was  observed  through  the 
questionnaire  for  “User  Interface  Satisfaction”  (Appendix  C).  This  questionnaire 
provided  subjective  but  quantitative  data,  which  are  summarized  in  Table  IV-2. 
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Question 

Participant 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

P1 

6.0 

5.0 

6.0 

7.0 

7.0 

8.0 

8.0 

7.0 

7.0 

6.0 

7.0 

7.0 

8.0 

8.0 

9.0 

8.0 

6.0 

6.0 

P2 

7.0 

8.0 

7.0 

7.0 

5.0 

6.0 

6.0 

7.0 

7.0 

7.0 

5.0 

6.0 

7.0 

7.0 

9.0 

9.0 

9.0 

5.0 

P3 

9.0 

7.0 

7.0 

8.0 

7.0 

9.0 

9.0 

7.0 

9.0 

7.0 

9.0 

9.0 

9.0 

9.0 

9.0 

9.0 

9.0 

7.0 

P4 

7.0 

7.0 

7.0 

7.0 

7.0 

8.0 

8.0 

8.0 

8.0 

7.0 

7.0 

7.0 

8.0 

8.0 

8.0 

8.0 

7.0 

8.0 

P5 

9.0 

8.0 

8.0 

5.0 

7.0 

9.0 

9.0 

7.0 

7.0 

8.0 

9.0 

9.0 

9.0 

7.0 

9.0 

9.0 

9.0 

7.0 

Mean 

7.6 

7.0 

7.0 

6.8 

6.6 

8.0 

8.0 

7.2 

7.6 

7.0 

7.4 

7.6 

8.2 

7.8 

8.8 

8.6 

8.0 

6.6 

Std  Dev. 

1.3 

1.2 

0.7 

1.1 

0.9 

1.2 

1.2 

0.4 

0.9 

0.7 

1.7 

1.3 

0.8 

0.8 

0.4 

0.5 

1.4 

1.1 

95%  Cl 

5.9 

5.5 

6.1 

5.4 

5.5 

6.5 

6.5 

6.6 

6.5 

6.1 

5.3 

5.9 

7.1 

6.8 

8.2 

7.9 

6.2 

5.2 

9.2 

8.5 

7.9 

8.1 

7.7 

9.5 

9.5 

7.7 

8.7 

7.9 

9.5 

9.2 

9.2 

8.8 

9.3 

9.2 

9.8 

8.0 

Table  IV-2  Questionnaire  Responses 


Table  IV-2  notes: 

•  Seores  go  from  0  to  9  —  from  worst  to  best. 

•  Questions  (see  Appendix  C  for  a  detailed  description): 

•  Overall  reaction  to  the  Event  Graph  Design  Tool  (questions  1  to  5) 

•  Screen: 

•  Characters  on  the  computer  screen  (question  6) 

•  Tool  bar  with  buttons  (question  7) 

•  Organization  of  information  on  screen  (question  8) 

•  Terminology  and  system  information: 

•  Computer  terminology  is  related  to  the  task  being  done 
(question  9) 

•  Menu  Items  and  Tool  bar  with  buttons  function 
identification  (question  10) 

•  Instruction 

•  Operation  of  the  system  (question  11) 

•  New  features  explored  by  trial  and  error  (question  12) 

•  Names  and  use  of  commands  are  learned  (question  13) 

•  Tasks  are  performed  in  a  straightforward  manner  (question 
14) 

•  System  capabilities 

•  System  speed  (question  15) 

•  System  reliability  (question  16) 

•  Correction  of  mistakes  (question  17) 

•  Experienced  and  inexperienced  users'  needs  are  taken  into 
consideration  (question  1 8) 
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Figure  IV-4  Participants’  Responses  for  Overall  Reaction  Questions 


Figure  IV-4  shows  the  participants  opinions  of  the  tool  (questions  1  to  5).  The 
questions  related  to  the  participants  overall  reaction  to  the  EGGDT.  The  questionnaire 
used  quantitative  scales  from  0  to  9  labeled  from  terrible  to  wonderful;  difficult  to  easy; 
frustrating  to  satisfying;  dull  to  stimulating;  rigid  to  flexible.  The  values  assigned  by  the 
participants  are  evaluated  as  “good”  considering  the  immature  state  of  the  tool.  However, 
participants  did  not  consider  the  tool  flexible.  It  was  assumed  that  this  immaturity  could 
be  eliminated  in  successive  versions  of  the  product. 

The  mean  for  question  1 8  (“Experienced  and  inexperienced  users'  needs  are  taken 
into  consideration”)  in  Table  IV-2  is  the  lowest  among  the  means.  Eigure  IV-5  shows  the 
individual  scores.  As  a  result  of  these  scores  more  emphasis  has  to  be  made  in  designing 
the  tool  so  that  it  takes  experience  into  consideration.  A  means  of  achieving  this  goal  is 
through  help  topics  and  a  self-study  courses.  However,  these  improvements  were  not 
planned  for  the  product’s  iteration  covered  by  this  thesis  research  effort. 


41 


Figure  IV-5  Participants’  Responses  for  “Experience  Taken  into  Consideration” 


In  conclusion,  the  users’  first  impression  was  good.  Participants  seemed  to  be 
concerned  about  adaptability  to  inexperienced  and  experienced  users  and  flexibility  of  the 
tool. 


3,  Qualitative  Data 

Throughout  each  session,  notes  were  taken  by  the  evaluator.  This  section  presents 
a  summary  of  that  qualitative  data. 

a.  Buttons  and  Menus 

•  The  buttons  “print”,  “attach  note”  “generate  Java”  and  “properties” 
were  difficult  to  identify. 

•  Menu  item  “Message  Bar”  should  be  called  “Status  Bar”. 

•  Most  participants  first  used  the  menu  items  to  do  the  tasks,  instead  of 
buttons. 

b.  Modality 

•  At  the  beginning  of  the  experiment,  participants  had  problems 
understanding  the  permanent  modality. 

•  Most  participants  did  not  identify  the  option  “Select  Element”,  which 
had  to  be  selected  to  move  events  and  reshape  edges. 

•  Most  participants  did  not  realize  that  changes  in  the  cursor  were 
associated  with  mode  changes. 

c.  Edges 

•  All  participants  had  problems  initially  creating  edges.  When  they 
noticed  the  status  bar  provided  help,  most  of  them  were  able  to  solve 
the  problem.  Eater  in  the  session,  they  had  no  problems  creating  edges. 
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•  When  edges  overlapped,  the  partieipants  had  diffieulties  in  seleeting 
and  editing  them. 

•  Some  partieipants  eomplained  that  the  inflection  points  could  not  be 
deleted. 

•  Participants  had  some  problems  moving  selection  markers  of  the 
inflection  points  of  the  edges,  especially  if  they  were  in  a  sharp  corner 
(programming  problem). 

d.  Text  Editing 

•  Participants  had  no  problems  editing  event  names;  however,  they  had 
problems  editing  edge  properties  when  edges  superimpose  one 
another. 

•  Some  participants  tried  to  exit  editing  using  the  Enter  key.  To  exit 
editing,  it  was  necessary  to  click  outside  the  editor  area. 

G.  LESSONS  LEARNED 

The  gathering  of  data  (times  and  errors)  by  the  experiment’s  controller  was  a 
difficult,  error-prone  task.  When  possible,  a  computer  application  must  manage  the 
collecting  of  data. 

The  questionnaire  was  too  dense.  Questions  have  to  be  selected  carefully  so  as  not 
to  confuse  participants.  For  example,  the  five  questions  about  the  overall  opinions  of  the 
tool  were  overwhelming  and  unnecessary. 

Pilot  experiments  are  the  key  to  discovering  flaws  in  the  design  of  experiments 
and  protocols.  The  two  pilot  experiments  discovered  important  defects  in  the  proposed 
design,  and  as  so,  they  helped  to  obtain  more  accurate  experimental  data  for  the 
evaluation. 

H.  CONCLUSION 

A  formative  experiment  was  planned  in  the  early  stages  of  EGGDT  development 
to  test  the  usability  of  the  approach  chosen  for  the  GUI.  The  usability  attributes  tested 
were  “initial  performance”,  “learnability”  and  “first  impression”. 

A  basic  prototype,  incorporating  all  interface  options  but  constraining  its 
functionality,  was  built.  After  two  pilot  experiments  that  uncovered  major  flaws  in  the 
experiment’s  design,  five  participants  performed  the  experiment  in  different  sessions.  The 
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experiment  eonsisted  of  a  battery  of  tasks  perform  with  the  prototype.  Eaeh  participant’s 
times  and  errors  were  recorded  along  with  their  responses  to  the  questionnaire. 

Major  problems  were  related  to  the  way  edges  were  created.  In  response,  a  new 
formative  experiment  was  implemented  to  compare  different  edge  construction  methods. 
This  experiment  is  discussed  in  the  next  chapter. 
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V.  SECOND  EXPERIMENT:  ARC  CONSTRUCTION 


A,  INTRODUCTION 

During  the  design  of  the  Event  Graph  Graphieal  Design  Tool  (EGGDT)  two 
formative  experiments  were  eonducted  to  ensure  that  the  final  product  would  fulfill  user 
expectancies.  The  first  formative  experiment  was  described  in  Chapter  IV.  The  second 
experiment^,  which  had  particular  concern  with  the  way  the  edges  are  built,  is  discussed 
in  this  chapter. 

B,  PURPOSE  OF  THE  EXPERIMENT 

After  the  first  experiment,  acceptance  of  the  EGGDT  was  clearly  predicated  on 
the  technique  employed  to  draw  edges.  The  first  prototype  provided  a  very  rudimentary 
tool  to  create  and  manipulate  edges.  Two  alternative  techniques  were  proposed;  the  direct 
or  free  draw  method  (E  method)  and  the  right  angle  or  vertical  horizontal  method  (VH 
method).  This  experiment  tried  to  determine  which  technique  was  more  appropriate  to  be 
employed  in  the  EGGDT. 

C,  DESIGN  OF  THE  EXPERIMENT 

A  crossover  experiment  design  was  selected  with  the  following  arrangement 


Ri 

Xi 

On 

X2  O12 

R2 

X2 

O21 

Xi  O22 

Selecting  experiment  participants  was  not  random  (N),  because  they  had  to  be 
students  of  the  Naval  Postgraduate  School  (NPS).  However,  from  the  initial  sample,  two 
groups  were  randomly  selected  (Ri  and  R2);  one  of  these  groups  used  the  VH  method  first 
(Xi)  and  then  the  E  method  (X2),  while  the  others  performed  the  experiment  in  the 
opposite  order  (E  and  then  VH).  Observations  of  both  treatments  were  taken  (Oi  and  O2). 


^  This  the  material  in  this  chapter  was  partially  developed  as  a  final  project  for  the  Human  Factors 
Course  OA3401  (Prof  Nita  Miller)  and  in  group  with  James  Campbell,  Parke  Paulson,  and  Erik  Hovda. 
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The  two  explanatory  variables  ehosen  were  the  differenee  in  time  required  to 
perform  the  related  tasks  with  eaeh  method  and  the  partieipants’  preference  between  the 
two  methods. 

To  design  the  experiment  the  following  validity  threats  were  considered. 

1.  Internal  Validity 

a.  History 

The  threat  to  validity  posed  by  history  (commonly  referred  to  as  the 
learning  effect)  was  addressed  by  a  crossover  experimental  design.  This  threat  is  a 
consequence  of  the  multiple  treatment  or  exposures  to  the  prototype.  For  example,  the 
first  time  the  participants  see  the  prototype,  they  are  naive  and  do  not  know  exactly  what 
to  expect.  The  second  time  the  participants  see  the  prototype,  they  have  expectations  that 
they  acquired  during  the  first  treatments.  Their  performance  could  thereby  be  affected  by 
the  order  in  which  the  exposures  to  the  prototype  occur.  Alternating  the  order  of  the 
treatments  for  each  participant  mitigates  this  problem. 

b.  Selection 

Participant  population  was  not  completely  random;  however,  to  help 
minimize  this  potential  threat,  the  order  in  which  each  participant  did  the  experiment 
tasks  was  random. 

2.  External  Validity 

This  threat  was  minimal  because  the  population  of  future  users  was  defined  as 
students  within  the  OR  curriculum.  The  sample  was  drawn  from  this  same  group. 

3.  Conclusion  Validity 

Selected  explanatory  variables  correspond  to  the  question  that  was  being  asked. 
The  participant  preference  and  the  time  to  complete  the  experiment  were  believed  to  be 
valid  variables  in  determining  which  technique  was  better. 
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4,  Statistical  Validity 

ANOVA  was  determined  to  be  the  best  ehoiee  to  analyze  all  of  the  gathered  data. 
Logistie  regression  was  eonsidered  as  a  statistieal  tool  that  eould  provide  further  insight 
into  the  data. 

D.  DESIGN  OF  THE  PROTOTYPE 

The  prototype  implemented  the  two  teehniques  to  draw  edges.  The  F  method 
allowed  drawing  straight  arrows  to  eonneet  eireles.  Dragging  in  any  internal  point 
reshaped  ares;  ares  eould  take  any  shape  desired  with  any  number  of  infleetion  points.  A 


Figure  V-1  The  Direet  or  Free  Method  (F  Method) 


The  VH  method  (Figure  V-2)  created  arcs  with  one  horizontal  and  two  vertical 
segments  that  could  be  moved  up  or  down  and  left  or  right.  Two  end  segments  (tail  and 
head)  connected  to  the  circles  and  could  be  rotated  at  any  angle  around  them.  The 
prototype  also  provided  two  selectable  points  in  the  junction  between  the  end  segments 
and  the  vertical  segments  that  could  be  moved  any  place. 
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Figure  V-2  The  Right  Angle  or  Vertieal  Horizontal  Method  (VH  Method) 

An  applieation  was  developed  to  hold  the  prototype  and  to  take  eontrol  of  the 


experiment’s  flow.  This  program  also  colleeted  the  data  from  eaeh  experiment. 

E.  EXPERIMENTAL  PROTOCOL 

Eighteen  partieipants  were  reeruited  to  be  subjeets  in  the  experiment  that  were 
under  the  eontrol  of  three  different  eontrollers.  The  Loosely  Coupled  Components  (LCC) 
laboratory  and  Human  Faetors  Human  Systems  Integration  Laboratory  (HSIL)  in 
Glasgow  Hall  at  NPS  were  used  to  set  up  the  experiments. 

Experiments  were  eontrolled  by  the  eomputer  program  referred  to  above. 
Experiment  eontrollers  had  to  explain  the  details  of  the  experiment  and  the  use  of  eaeh 
method,  while  doeumenting  the  partieipant’s  times  for  eaeh  task  and  number  of  errors 
ealeulated  by  the  applieation.  This  kept  the  influenee  of  the  eontrollers  at  a  minimum. 
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Figure  V-3  Welcome  Window 


The  Figure  V-3  shows  the  welcome  window. 


‘  EGGDT  Edge  Mode  Enperiment  —  Working  Area 


^2^ 


^  EGGDT  Edge  Mode  EKperiment 


TREATMENT  A  -  TRAINING 


This  Arc  mode  follows  a  Vertical/Horizontal  pattern 
It's  possible  to  reshape  arcs: 

Vertical  segments  can  be  dragged  along  the  X  axis  —  The  horizontal  segment  can  be  dragged  along  the  Y  axis  —  End  segments  can  be  dragged  around  the  circles,  maintaining  thi 
Selected  Points  can  be  dragged  to  any  place  inside  the  working  area  —  To  delete  an  arc  press  the  DELETE  key  while  the  arc  is  selected 
Try  to  create  and  arc  from  1  to  6  by  dragging  from  1  and  releasing  over  6,  and  then  try  to  reshaped 

Play  creating  and  reshaping  ars  until  you  feel  confortable.  You  can  get  help  at  this  point  from  the  experiment  controler  —  When  you  finish  press  DONE  to  start  the  experiment 


iHstartlll  23  ^  V  Cjl  ”ll  I  ^lnbQ.,.|  aH:\T...|  g]Anal.,.|  B]Expe...|  8]poa...|  BCiV,.  |  Bjthesi...|  §  EG...  ||^  EG...  |<[  §0 


Figure  V-4  Training  Window 


49 


Experiments  were  divided  into  four  phases.  In  the  first  phase  partieipants  learned 
how  to  use  one  of  the  methods.  As  diseussed  above,  the  order  in  whieh  the  partieipant 
was  given  the  methods  was  random.  Figure  V-4  and  Figure  V-5  show  the  VH  are 
eonstruetion  method’s  training  and  task  window.  Partieipants  had  the  opportunity  to  play 
with  this  teehnique  in  the  training  window.  In  the  next  step,  they  had  to  eomplete  all  tasks 
in  the  task  window  and  press  the  done  button.  The  system  displayed  a  window  with  the 
results  (Figure  V-6). 


Start  I  ]  ^  ^  ^  ”11  |^Inbox-Mi,..|  |  ij^JBuilder  |  •'^[8]01;36  ...l  ^Institutio,,.  |  ^Applicatio.,,||^  EGGDT...'  ^  EGGDTE,,.|  11:05  AM 

Figure  V-5  Task  Window 

The  procedure  to  perform  the  experiment  with  the  other  treatment  was  the  same. 
The  tasks  to  perform  were  mirrored  to  the  ones  done  first,  for  example  if  one  task  was  to 
create  an  edge  between  events  0  and  7  and  other  between  events  5  and  2;  the  mirror  tasks 
were  create  edges  between  3  and  4  and  between  6  and  5. 
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Figure  V-6  Result  window 

Appendix  D  inventories  the  data  collected. 

F.  STATISTICAL  ANALYSIS 

As  stated  above,  the  objective  of  the  experiment  was  to  determine  which 
alternative  arc  technique  was  preferred.  Two  factors  influenced  the  selection;  the 
difference  in  time  to  perform  the  tasks  with  the  two  different  methods  and  personal 
preference.  This  statistical  study  demonstrates  that  the  F  method  was  preferred  over  the 
VH  method. 

1,  Difference  in  Performance  Time 

The  scatter  plot  in  Figure  V-7  shows  the  relationship  between  test  times  for  each 
individual  participant.  Points  plotted  above  the  line  show  better  time  performance  on  the 
VH  method  of  arc  construction.  Evidence  demonstrated  that  all  but  four  participants 
scored  better  in  the  F  method. 
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Figure  V-7  Relationship  Between  Test  Times  per  Partieipant 


The  following  ANOVA  (Table  V-1)  shows  a  significant  difference  between  the 
two  times  to  complete  the  tasks  (p-value  of  .032). 


Df 

Sum  of  Sq 

Mean  Sq 

F  Value 

Pr(F) 

VH  Time 

16 

10446.29 

10446.29 

5.59 

0.032 

Table  V-1  ANOVA  Results  of  Time  to  Perform  Tasks  with  F  Method  vs.  Time  with 

VH  Method 

This  above  analysis  indicates  that  the  F  Method  provides  a  faster  way  to  draw 
arcs.  However,  a  linear  model  was  built  to  test  if  other  factors  influenced  this  difference 
in  performance. 

2,  Linear  Model  to  Explain  Time  Difference 

The  only  linear  combination  that  could  be  fitted  to  explain  time  difference  was 
“Order”  (VH/F  or  FA^H)  plus  “self-expressed  experience  with  graphical  environments” 
(like  PowerPoint)  plus  “self-expressed  level  in  computers  use”. 
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Coefficients 

Value 

Std,  Error 

t  value 

Pr(>|t|) 

Intercept 

120.03 

47.56 

2.52 

0.020 

Order 

36.33 

9.84 

3.69 

0.002 

Graphical 

63.76 

16.43 

3.88 

0.002 

Level 

-93.42 

22.37 

-4.18 

0.001 

Residual  stand.  Error 

36.03  on  14  degrees  of  freedom 

Multiple  R-Squared 

0.60 

F-statistic 

7.17  on  3  and  14  df,  p-value  is  0.004 

Table  V-2  Linear  Model  Summary.  Difference  of  Time  Performance  Against  Order, 

Graphical  Experience  And  Computer  level 


Table  V-2  summarizes  this  linear  model,  which  presents  several  problems; 

•  The  “Level’s”  coefficient  value  is  negative  but  the  “Graphical”  experience 
is  positive.  This  sign  difference  is  confusing,  since  these  two  coefficients 
had  been  considered  positively  correlated. 

•  The  R  squared  is  0.60,  therefore,  much  of  the  variability  in  time  difference 
is  not  explained  with  this  model.  It  is  assumed  that  this  variability  is  due  to 
the  different  characteristics  of  the  two  methods. 

•  The  large  residual  standard  error  36.03  is  an  indication  that  this  is  not  a 
good  model. 

In  conclusion,  although  it  is  possible  to  fit  a  linear  model  (a  bad  one)  to  explain 
the  time  performance  difference,  the  difference  in  time  is  explained  by  the  contrast 
between  techniques  and  not  by  other  factors. 

3,  User  Preference 

The  pie  chart  in  Figure  V-8  gives  a  summary  of  the  participants’  preferences. 


Pie  Chart  -  User  Preference 


Figure  V-8  Preference  of  the  Participants  (VH  or  F) 
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Clearly  the  participants  expressed  a  preference  for  the  F  Method.  However  to  test 
if  other  factors  influenced  this  preference,  the  following  analysis  was  conducted. 

4,  Relationship  Between  Preference,  Performance  Time  and  Treatment 
Order, 

Figure  V-9  displays  the  individual  performance  time  differences  conditioned  by 
the  expressed  preference  (VH  or  F).  Only  one  person  who  preferred  the  VH  method 
actually  scored  better  in  the  vertical-horizontal  time.  On  the  other  hand,  as  many  as  four 
participants  scoring  better  in  VH  preferred  the  F  method.  This  indicates  that  test  times 
were  not  a  factor  in  determining  a  participant’s  preference. 


Dot  plot.  Difference  between  VH  and  F  times  conditioned  in  Preference 

VH 
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Figure  V-9  Individual  Performance  Time  Differences  Conditioned  by  the  Preference 

(VH  or  F) 


Logistic  Regression  was  performed  to  test  if  the  preference  had  any  relationship 
with  the  treatment  order  or  performance  times.  Table  V-3  presents  the  summary  of  the 
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regression.  The  p-values  show  that  the  order  of  the  treatments  (VH/F  or  FA^H)  and  the 
performanee  times  were  not  signifieant  to  the  expressed  preferenee. 


Coeffieients 

Value 

Std.  Error 

T  value 

p-values 

Intereept 

3.90 

2.90 

1.30 

Order 

-2.35 

1.57 

-1.49 

0.13 

VH  time 

-0.05 

0.03 

-1.59 

0.11 

E  time 

0.02 

0.03 

0.77 

0.29 

Table  V-3  Logistie  Regression  Summary.  Preferenee  Against  Order  and 

Performanee  Times 


In  eonelusion,  the  partieipants’  preferenee  did  not  seem  to  be  affeeted  by  any 
external  faetor.  Extrapolating  to  the  entire  EGGDT  population,  this  result  indieates  that 
the  E  method  is  eonsidered  better  by  the  potential  users  of  the  tool. 

G.  CONCLUSION 

A  seeond  formative  experiment  was  performed  to  test  the  best  method  to  draw 
edges  in  the  EGGDT.  A  erossover  experimental  design  helped  to  relieve  the  history  threat 
that  was  identified  for  this  experiment.  The  explanatory  variables  ehosen  were  the 
differenee  in  time  to  perform  the  related  tasks  with  eaeh  method  and  the  individual 
preferenee. 

The  prototype  implemented  the  VH  and  E  method.  An  applieation  eontained  the 
prototype  and  provided  aeeounting  of  tasks  times  and  errors.  This  applieation  also 
provided  help  for  both  methods.  The  partieipants  had  the  opportunity  to  praetiee  eaeh 
method  in  a  speeifie  training  window  before  performing  the  aetual  tasks  of  the 
experiment.  Nineteen  partieipants  eollaborated  in  the  experiment  in  separate  sessions  that 
were  supervised  by  three  experiment  eontrollers. 

Based  on  the  responses  from  the  preferenee  survey,  as  well  as  the  lower  times  for 
the  free  draw  are  eonstruetion  method,  and  the  eonelusions  of  the  statistieal  study  above, 
the  Tree  Draw  Are  Method  (E  Method)  was  eoneluded  to  be  the  best  ehoiee  for  eontinued 
use  and  development  of  the  EGGDT. 
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VI.  FINAL  EXPERIMENT:  USERS’  JUDGMENTS  TOWARD 

GRAPHICAL  DESIGN  TOOLS 


A.  INTRODUCTION 

As  stated  in  Chapter  I,  this  thesis  evaluates  whether  the  use  of  graphical  design 
tools  improves  the  satisfaction  of  OR  analysts  developing  simulation  models.  To  test  this 
hypothesis,  a  final  experiment  was  performed  using  Version  0.3  of  the  Event  Graph 
Graphical  Design  Tool  (EGGDT).  This  chapter  summarizes  the  experiment  and  analyzes 
the  results.  A  description  of  the  capabilities  of  the  EGGDT  Version  0.3  is  also  provided 
in  paragraph  D. 

B.  PURPOSE  OF  THE  EXPERIMENT 

The  purpose  of  this  experiment  was  to  determine  if  the  use  of  the  EGGDT,  or  any 
other  graphical  design  tool,  increases  OR  analyst  satisfaction  when  developing  simulation 
applications. 

C.  DESIGN  OF  THE  EXPERIMENT 

The  experiment  was  designed  as  follows: 

-KT  Ri  Xi  O  01. ..9 

R2  X2  O10...19 

Participants  were  recruited  from  the  population  of  NPS  students  (see  Chapter  III) 
so  the  initial  selection  was  not  random  (N).  Two  random  samples  of  the  selected  users 
(Ri  and  R2)  were  obtained  to  perform  two  sessions  (Xi  and  X2).  Finally,  all  participants 
fdled  out  an  individual  questionnaire  (Oi). 

1.  Explanatory  Variables 

Initially  the  explanatory  variables  hypotheses  tested  were: 

•  Participants’  prior  opinion  of  these  kinds  of  tools  is  related  to  user 
satisfaction. 

•  Participants’  preference  for  using  manual  versus  computer-aided  tool  is 
related  to  user  satisfaction  with  the  tool. 

The  possible  answers  to  these  questions  were  “yes”,  “no”  or  “no  opinion”.  After  a 

pilot  experiment,  it  became  clear  that  answers  to  these  two  questions  could  not  account 
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for  differences  since  participants’  responses  were  all  the  same.  For  this  reason,  it  was 
decided  to  include  two  more  explanatory  variables  in  the  questionnaire; 

•  Participants’  perception  of  how  the  use  of  these  tools  can  influence  the  OR 
field. 

•  Participants’  feelings  toward  simulation  after  the  session. 

The  choices  in  these  questions  were  arranged  from  1  to  5  using  traditional  Likert 
scaling.  Likert  scaling  of  responses  allows  for  use  of  parametric  statistics  in  the  analysis. 

2,  Participants 

The  cooperation  of  nineteen  participants  in  two  evaluation  sessions  was  obtained. 
These  participants  were  representative  of  the  group  of  users  of  the  EGGDT  as  defined  in 
Chapter  III.  The  volunteers  were  OR  students  at  the  Naval  Postgraduate  School  (NPS)  in 
either  their  fourth  or  eighth  quarter  of  school. 

D.  PROTOTYPE  DESIGN 

Version  0.3  of  the  EGGDT  was  the  final  prototype  developed  for  this  thesis 
research,  and  the  experiments  covered  in  this  chapter  were  performed  with  this  program. 
The  prototype  implements  the  following  features: 

•  Allow  the  creation  of  EGs  and  all  their  components  (event,  edges  and  simulation 
variables). 

•  Allow  the  drawing  of  EGs  as  graphical  pictures  as  stated  in  Chapter  III. 

•  Allow  the  deletion  and  modification  of  elements  of  EGs. 

•  Provide  the  functions  “Save”,  “Save  As”,  “Open”  and  “Rename”. 

•  Generate  the  Java  class  matching  the  EG. 

When  the  prototype  starts,  it  shows  a  choice  dialog  in  Eigure  VI- 1.  The  user  can 
choose  to  create  a  new  EG  or  open  an  existing  one. 


Eigure  VI- 1  Initial  Choice  Dialog  of  the  EGGDT 
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If  the  “Create  New  EG”  option  is  selected,  the  EGGDT  shows  the  EG  properties 
window  in  Eigure  VI-2.  This  version  of  the  EGGDT  only  implemented  the  “Name”  and 
“Directory”  properties;  the  rest  of  the  fields  are  considered  informative.  Eor  example,  if 
the  user  provides  a  project  name,  the  EGGDT  does  not  create  a  project. 
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«  PiUMBittes  Window 


fProperties  [  Directories  | 

Name  arrivals 

Project  EGGDT  Evaluation  Phase 

Package 

Title  I 
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Author  Angel  San  Jose 
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Copyright  NPS 


Description  Basic  Arrival  System 


Gioph  Piupniii.i  Window 
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Figure  VI-2  Properties  Window 
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Figure  VI-3  Snapshot  of  the  EG  for  the  “Arrival  Model” 
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Elements  of  EGs  (events,  edges  and  simulation  variables)  can  be  created  with  the 
tool.  An  example  of  a  “Arrival  Model”  is  in  Figure  VI-3.  The  right  side  of  the  EGGDT  is 
used  to  draw  the  graphical  elements  that  represent  events  and  edges. 

The  upper-left  side  contained  the  state  variables  and  simulation  parameters.  The 
lower-left  side  showed  the  properties  of  the  selected  element.  Properties  of  events 
included  name,  actions  and  input  parameters.  Properties  of  edges  convey  delay  time, 
condition  and  arguments  matching  the  input  parameters  of  the  target  event. 


Figure  VI-4  Snapshot  of  the  Java  Class  Generated  Code  Window  for  the  “Arrival 

Model” 


The  EGGDT  generates  the  Java  class  source  code  on  demand  by  pressing  the  L^LI 
button.  The  source  code  window  is  shown  in  Figure  VI -4.  The  source  code  window  is  not 
editable,  but  it  can  be  saved,  edited,  compiled  and  run  outside  the  EGGDT  program. 


The  generated  source  code  is  not  actually  ready  to  be  compiled,  but  it  constitutes 
a  skeleton  of  a  Java  program.  It  also  provides  the  “get  method”  for  state  variables  and 
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simulation  parameters  and  “set  method”  for  simulation  parameters.  Table  VI- 1  shows  the 
eomplete  Java  elass  generated  from  the  “Arrivals”  model  (header  eomments  have  been 
suppressed). 


import  simkit.*; 

public  class  arrivals  extends  SimEntityBase  { 

//  State  Variables 

private  int  waitingLineLength  =  0  ;  //  Sym  =  q 
private  int  numEntered  =  0  ;  //  Sym  =  n 
private  int  serversAvail  =  numServers  ;  // 
Symbol  =  s 

//  Simulation  Variables 

private  int  maxNumToEnter  =  10000  ;  // 

Symbol  =  maxN 

private  int  numServers  =  3  ;  //  Symbol  =  nS 
private  Exponential  arrivalTime  ;  //  Symbol  = 
ta 

private  String  serviceTime  ;  //  Symbol  =  ts 

public  void  doRun  ()  { 
waitDelayC'Arrival'' ,  ta ); 

} 

public  void  doArrival  ()  { 

n+-l-; 

q++; 

if(  n  <  maxN  )  { 
waitDelayC'Arrival"  ,  ta ); 

} 

if  (  s  >  0  )  { 

waitDelayC'StartService"  ,  0.0  ); 

} 

} 

public  void  doStartService  ()  { 

q-s 

S-; 

waitDelayC'EndService"  ,  ts  ); 

} 

public  void  doEndService  ()  { 

S++; 

if  (  q  >  0  )  { 

waitDelayC'StartService"  ,  0.0  ); 

} 


//  ***  SETTERS  &  GETTERS  *** 
public  int  getWaitingEineEength( )  { 
return  waitingEineEength  ; 

} 

public  int  getNumEntered( )  { 
return  numEntered ; 

} 

public  int  getServersAvail( )  { 
return  serversAvail ; 

} 

public  int  getMaxNumToEnter( )  { 
return  maxNumToEnter ; 

} 

public  int  getNumServers( )  { 
return  numServers  ; 

} 

public  Exponential  getArrivalTime( )  { 
return  arrivalTime  ; 

} 

public  String  getServiceTime( )  { 
return  serviceTime ; 

} 

public  void  setMaxNumToEnter(  int 
maxNumToEnter)  { 

this.  maxNumToEnter  = 

maxNumToEnter ; 

} 

public  void  setNumServers(  int 
numServers)  { 

this.numServers  =  numServers  ; 

} 

public  void  setArrivalTime( 
Exponential  arrivalTime)  { 

this. arrivalTime  =  arrivalTime  ; 

} 

public  void  setServiceTime(  String 
serviceTime)  { 

this. serviceTime  =  serviceTime  ; 

} 

} 


Table  VI- 1  Java  Class  Generated  from  the  EG  for  the  “Arrivals  Model” 
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E.  EXPERIMENTAL  PROTOCOL 

The  participants  were  gathered  in  the  student  laboratory,  GL-318,  in  Glasgow 
Hall  at  the  Naval  Postgraduate  School  (NPS)  on  August  24  and  28,  2001.  The  principal 
investigator  explained  the  purpose  of  the  experiment  and  then  passed  out  the  informed 
consent  form  in  Appendix  E  for  the  participants  to  read  and  sign. 


Figure  VI-5  Snapshot  of  EG  for  the  “CPU  Modef’ 

A  quick  tour  through  the  tool  was  performed  in  a  lecture  fashion.  The  principal 
investigator  explained  how  to  use  the  EGGDT  and  the  participants  could  follow  the  tasks 
on  their  computers.  A  snapshot  of  the  example  used  to  illustrate  the  capabilities  of  the 
EGGDT  is  in  Figure  VI-5.  Participants  were  also  allowed  to  ask  questions  about  the  tool. 
The  principal  investigator  explained  the  possibilities  of  future  versions  of  the  EGGDT. 
Finally,  participants  filled  out  the  questionnaires. 
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F. 


STATISTICAL  ANALYSIS 


Analysis  of  the  questionnaire  data  should  that  all  partieipants  felt  that  the  EGGDT 
and  similar  tools  eould  improve  their  satisfaction  when  developing  simulation 
applications.  Participants  also  expressed  their  unanimous  preference  for  using  these  tools 
rather  than  manual  methods  to  develop  simulation  models. 


Figure  VI-6  Participants’  Responses  to  Questions  II. C  and  II.D 


Figure  VI-6  shows  participants’  responses  to  questions  II. C  and  II.D  (see 
questionnaire  in  Appendix  F) 

•  II. C.  -  After  viewing  this  presentation,  how  useful  do  you  think  these  tools  will  be 
in  the  OR  field?  (briefly,  “Usefulness  of  the  Tools”). 

•  II.D.  -  Assuming  you  were  skeptical  about  using  simulation  in  your  future  OR 
projects,  how  would  you  rate  your  feelings  about  simulation  after  having  seen  this 
experiment?  (briefly,  “Feelings  toward  Simulation  after”.) 

The  choices  for  the  question  “Usefulness  of  the  Tools”  were; 

1 .  These  tools  are  not  needed. 
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2.  These  tools  are  valid  but  not  needed. 

3.  These  tools  eould  have  limited  utility  in  speeifie  applieations. 

4.  These  tools  have  utility  in  a  wide  range  of  applieations. 

5.  These  tools  are  a  major  step  forward  in  the  OR  field. 

Thirteen  out  of  nineteen  partieipants  thought  the  tool  has  utility  in  a  wide  range  of 
applieations;  four  partieipants  eonsider  the  utility  limited  to  speeifie  applieations, 
whereas,  two  of  them  expressed  their  belief  that  this  tool  is  a  major  step  forward  in  the 
OR  field.  These  responses  eorroborate  that  partieipants  were  exeited  about  these  kinds  of 
tools. 

The  ehoiees  for  the  question  “Feelings  toward  Simulation  After”  were: 

1 .  My  feelings  are  unehanged;  I  probably  would  not  use  simulation. 

2.  These  tools  seem  useful  but  I  would  only  use  them  as  a  last  resort. 

3.  With  a  tool  like  this,  I  think  I  eould  design  some  useful  simulations  with 
confidenee. 

4.  I  am  very  eonfidenee  that  these  tools  will  cause  me  to  favor  simulation  in 
most  OR  projects. 

5.  I  now  believe  that  simulation  should  be  my  primary  OR  tool. 

Fourteen  out  of  nineteen  participants  chose  answer  number  three.  The  remaining 
five  participants  chose  answer  number  four.  Participants  were  realistic  about  the 
possibilities  of  these  tools;  they  appreciated  the  usefulness  of  these  tools  but  were  aware 
of  their  limitations. 

The  two  questions  received  uniformly  “good”  grades.  Figure  VI-6  clearly  shows 
that  the  variables  were  positive  correlated.  Each  participant  assigned  the  same  or  better 
grade  to  the  question  “Usefulness  of  the  Tools”  than  to  the  question  “Feelings  toward 
Simulation  After”.  The  actual  correlation  value  is  0.547. 

Table  VI-2  summarizes  the  statistics  for  these  two  questions.  The  mean  for 
“Usefulness  of  the  Tools”  (3.9)  is  slightly  better  than  mean  for  “Feelings  toward 
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Simulation  After”  (3.2).  The  standard  deviations  were  very  small  (0.57  and  0.45),  that 
indieates  what  partieipants  agreed  with  these  questions. 

No  external  faetors  seem  to  influenee  the  partieipants’  responses.  The  following 
two  paragraphs  diseuss  the  partieipants’  responses  to  the  eomputer  profieieney  and 
demographie  questions. 


Usefulness 

of  the  Tools 

Feelings  toward 
Simulation 

After 

Min 

3 

3 

1  Quartile 

4 

3 

Mean 

3.9 

3.2 

Median 

4 

3 

3'^''  Quartile 

4 

3.5 

Max 

5 

4 

Total  # 

19 

19 

Std.  Dev. 

0.57 

0.45 

Table  VI-2  Summary  Statistics  for  Questions  II. C  and  II. D. 

1,  Computer  Proficiency  Questions 

Participants  were  questioned  about  their  opinions  in  simulation  before  and  after 
the  experiment.  Figure  VI-7  show  the  participants’  responses  to  question  II.D  “Feelings 
toward  Simulation  After”  versus  participant’s  responses  to  question  I.C  “Feelings  toward 
Simulation  Before”  the  experiment.  The  scales  in  this  graph  have  to  be  considered  with 
caution  because  the  multiple-choices  used  for  both  questions  were  different.  The  options 
for  “Feelings  toward  Simulation  After”  were  described  above;  the  options  for  “Feelings 
toward  Simulation  Before”  were: 

1 .  I  should  have  enrolled  in  the  IT  curriculum. 

2.  I  could  take  them  or  leave  them.  I  won’t  use  it  again. 

3.  I  think  it  can  help  to  do  some  pretty  cool  stuff. 

4.  I  like  simulation 

5.  I  think  I  going  to  focus  my  professional  future  in  this  area. 
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Most  participants  liked  simulation  before  the  experiment  and  they  thought  that 
tools  like  the  EGGDT  gave  them  more  confidenee  to  develop  simulation.  In  conelusion, 
the  future  OR  analysts  are  willing  to  use  simulation  in  their  projeets  and  think  that  these 
kinds  of  tools  ean  help  them  to  do  their  jobs. 


Figure  VI-7  Participants’  responses  to  questions  I.C  and  II. D. 

2,  Demographical  Questions 

The  data  gathered  from  the  demographieal  questions  of  the  questionnaire 
(questions  part  III  Appendix  E)  did  not  eontribute  to  explaining  the  study  results. 


66 


5 

4 

3 


5 

4 

3 


Figure  VI-8  Participants’  Responses  to  Question  “Usefulness  of  the  Tools” 

Conditioned  by  Military  Branch 
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As  an  illustration,  consider  the  graph  in  Figure  VI-8  that  shows  the  responses  to 
“Usefulness  of  the  Tools”  question  conditioned  by  military  branch;  no  pattern  of 
relationship  between  military  branches  and  “Usefulness  of  the  Tools”  was  identified. 

G.  CONCLUSION 

The  thesis  stated  in  Chapter  I  was  that  the  use  of  simulation-design  tools  can 
improve  OR  analyst  satisfaction  when  developing  simulation  models.  A  summative 
experiment  was  performed  to  prove  this  claim.  For  this  experiment,  Version  0.3  of 
EGGDT  was  used  to  illustrate  what  the  simulation-design  tools  could  do  for  OR  analysts. 

The  EGGDT  Version  0.3  implemented  the  most  important  requirements  stated  in 
Chapter  III  and  Appendix  B.  It  allowed  the  user  to  see  a  depiction  of  EGs  with  circles  and 
arcs  used  to  represent  events  and  edges.  It  generated  Java  code  from  EG  models.  Einally, 
it  offered  the  typical  file  management  options,  “Save”,  “Save  As”,  “New”,  “Open”  and 
“Rename”. 

The  experiment  was  performed  in  two  sessions.  The  experiment  controller  briefed 
the  tool  for  45  minutes  and  participants  filled  out  a  questionnaire.  The  data  gathered  from 
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these  questionnaires  elearly  shows  that  partieipants  believed  that  these  kinds  of  tools  can 
improve  user  satisfaction  and  they  expressed  their  preference  for  using  these  tools. 
Participants  stated  that  these  kinds  of  tools  are  very  useful  in  a  wide  range  of  OR 
applications  and  that  these  tools  increase  their  confidence  to  develop  simulation  models. 

In  conclusion,  from  the  data  collected  from  the  experiments,  OR  students  believe 
that  simulation-design  tools  improve  OR  analyst  satisfaction  when  developing 
simulations. 
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VII.  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  CONCLUSIONS 

This  thesis  demonstrated  that  the  use  of  simulation-design  tools  improved  the 
satisfaetion  of  the  Operation  Researeh  (OR)  analysts  while  developing  simulations  to 
model  OR  problems.  To  test  this  statement  it  was  neeessary  to  provide  a  simulation- 
design  tool  to  be  used  in  the  evaluative  experiments.  The  Event  Graph  Graphieal  Design 
Tool  (EGGDT)  was  developed  beeause  no  other  suitable  tool  was  available.  The  EGGDT 
is  a  Computer  Aided  Software  Design  (CASE)  program  to  develop  Diserete  Event 
Simulation  (DES)  models  using  Event  Graphs  (EG)  and  to  generate  the  Java  souree  eode. 
The  EGGDT  depiets  EGs  using  eireles,  ares  and  tables  for  representation  of  events,  edges 
and  simulation  variables. 

To  doeument  the  Analysis  and  Design  (A&D)  of  the  EGGDT,  the  Unified 
Modeling  Eanguage  (UME)  was  used.  UME  provided  a  way  to  eommunieate  A&D 
deeisions.  The  UME  artifaets  utilized  were  Use-Case  Diagrams,  Coneeptual  Model 
Diagrams,  Class  Diagrams,  Collaboration  Diagrams  and  State  Maehines.  Java  was 
ehosen  to  implement  the  EGGDT  beeause  Java  has  the  required  graphieal  eapabilities  and 
is  a  modern  objeet-oriented  multi-platform  language. 

Human  Eaetors  teehniques  were  used  to  evaluate  the  EGGDT.  The  User-Centered 
approaeh  was  eonsidered  an  appropriate  paradigm.  Two  parallel  software  development 
flows  were  used,  one  for  the  applieation  software  and  the  other  for  the  Graphieal  User 
Interfaee  (GUI)  software,  allowed  the  separation  between  applieation  domain  and 
interfaee  domain.  This  way  deeisions  related  to  the  GUI  were  not  affeeted  by  applieation 
details  leading  to  a  more  deeoupled  interfaee  design.  Using  prototypieal  versions  of  the 
EGGDT,  two  formative  experiments  were  eondueted  to  obtain  feedbaek  from  users 
during  the  EGGDT  implementation. 

The  first  formative  experiment  evaluated  the  initial  approaeh  ehosen  for  the  GUI 
of  the  EGGDT.  The  prototype  implemented  the  more  important  GUI  serviees,  but  it  did 
not  provide  the  related  funetionalities.  Eive  students  from  the  OR  eurrieulum  were 
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selected  to  perform  a  series  of  benchmark  tasks  with  the  prototype  and  to  answer  a 
questionnaire.  The  participants  expressed  their  positive  expectations  about  the  tool.  The 
benchmark  tasks  statistically  demonstrated  that  the  GUI  of  the  prototype  had  a 
satisfactory  leamability  level.  The  initial  design  of  the  GUI  of  the  EGGDT  was 
considered  valid;  however,  since  participants  encountered  problems  with  the  arc 
construction  method,  a  second  formative  experiment  was  planned  to  test  this  particular 
issue. 

The  second  experiment  focused  on  the  arc  construction.  Two  methods  were 
proposed  and  considered  by  the  users.  The  experiment  consisted  of  two  similar  batteries 
of  tasks.  Participants  had  to  perform  a  given  task  battery  with  each  method.  Nineteen 
participants  were  recruited  for  the  individual  sessions.  Times  and  errors  were  recorded 
and  user  preference  were  determined.  Participants  expressed  their  preference  for  the 
“Free  Method”  that  allowed  the  drawing  of  arcs  by  specifying  the  end  circles  and  any 
number  of  middle  points.  The  benchmark  times  showed  that  participants  did  their  tasks 
faster  with  the  “Free  Method”  than  they  did  with  the  other  one.  The  “Free  Method”  was 
then  selected  to  implement  the  FGGDT  version  0.3  used  for  the  summative  experiment. 

To  test  the  claim  that  simulation-design  tools  help  OR  analysts  who  are 
developing  simulation  models,  a  summative  experiment  was  performed  using  Version  0.3 
of  the  EGGDT  (which  was  the  last  version  implemented  during  this  thesis  research).  The 
experiment  was  based  on  a  questionnaire  that  was  filled  out  by  participants  after  the 
principal  investigator  briefed  the  EGGDT  as  an  example  of  a  simulation-design  tool.  Two 
sessions  were  completed  in  one  of  the  computer  laboratories  in  Glasgow  Hall  at  the 
Naval  Postgraduate  School  (NPS).  The  nineteen  participants  agreed  that  these  kinds  of 
tools  could  improve  their  satisfaction  in  developing  simulations;  they  also  unanimously 
stated  their  preference  for  using  these  tools  instead  of  manual  methods.  In  addition,  they 
expressed  their  opinion  that  these  tools  could  be  useful  in  a  wide  range  of  OR 
applications.  Finally,  they  stated  that  tools  like  the  EGGDT  encouraged  them  to  have 
more  confidence  when  facing  simulation  projects. 

In  conclusion,  this  investigation  shows  that  OR  students  at  the  NPS  consider  the 

EGGDT  and  similar  tools  an  essential  instrument  when  developing  simulations.  The 

70 


principal  investigator  also  believes  that  this  statement  can  be  generalized  to  all  OR 
analysts  and  to  others  involved  in  modeling  and  simulation  projects  and  studies. 

B,  RECOMMENDATIONS 

The  EGGDT  is  still  in  a  very  immature  state;  further  development  is  neeessary. 
The  next  iteration  of  the  EGGDT  development  ean  include  some  the  following: 

•  Refaetor  of  the  EGGDT  design,  eode  and  doeumentation. 

•  Implement  eaneeling  edges. 

•  Inelude  the  “Undo”,  Redo”,  “Copy”  and  “Paste”  serviees. 

•  Implement  a  serviee  to  save  EG  models  in  XML  format  (version  0.3  of  the 
EGGDT  saves  EG  model  in  binary  format). 

•  Improve  the  Java  eode  generator. 

•  Design  a  projeet  eontrol  approaeh  that  allows  the  ereation  of  multiple  related  EGs. 

•  Define  a  eomplete  methodology  that  ineludes  the  neeessary  phases  to  develop  a 
simulation  model,  beginning  with  the  requirements  specifieation.  The  A&D 
phases  eould  be  driven  by  UML  artifaets.  The  detail  design  of  simulation  classes 
has  to  involve  building  their  EGs. 

•  Implement  a  reengineering  tool  that  allows  the  extraetion  of  EG  models  from 
souree  eode  of  simulation  Java  elasses  implementing  the  Simkit  interfaees. 

The  use  of  the  EGGDT  by  the  OR  students  at  the  NPS  is  an  essential  means  of 
getting  input  from  real  users.  The  program  can  be  run  from  the  eommon  drive  of  the  OR 
department  network  and  it  is  ready  for  downloading  from  the  web  side  of  NPS  Loosely 
Coupled  Components  group  (http://diana.gl.nps.navy.mil/LCC/)  whieh  is  under  the 
supervision  of  Professor  Arnold  Buss. 
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APPENDIX  A.  EG  ELEMENTS 


A.  EG  ELEMENTS 


Element 

Representation/  Example 

Definition/  Observations 

State  Variables 

Alphanumeric  String. 

delayTimeInQueue,  numberInQueuel 

Snapshot  of  the  state  of  the  simulation. 

Events 

/  Event  \ 

1  Name  ) 

Y  (Param)  J 

Events  represent  a  particular  state  transition  in  the  system. 
[Buss96] 

Edges 

Liiii£(aig]f )  [cond] 

Represent  the  scheduling  of  an  event  from  other  event.  Two 
types: 

-  Scheduling  Edges; 

-  Canceling  edges,  cancels  the  next  occurrence  of  the  event. 
(Dashed  line) 

Attributes  (all  optional): 

-  Delay  time  to  schedule  the  target  event.  If  Delay  time  is  not 
present  it  is  considered  zero.  It  does  not  apply  to  Canceling 
Edges. 

-  Arguments  for  the  target  event. 

-  Condition  to  schedule  the  target  event. 

V  V 

1  ' 

(argj)  [coTiid] 

Simulation 

Parameters 

Alphanumeric  String. 

numOfServers, 

Simulation  Parameters  are  constant  during  a  simulation  run. 

Event  Actions 

Expressions 

{  Q++  ;  queue. add(customer) } 

Sequence  of  operations  (command)  that  are  executed  when  the 
event  is  activated 
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Delay  Times 

A  number  or  acronym  (t  plus  initial  of  the  event 
name) 

ta,  ts,  1.5 

Expressed  in  time  units.  The  units  have  to  be  uniform  across 
the  simulation. 

Edge  Conditions 

A  condition  expression  in  square  brackets. 

rs>oi 

The  event  is  scheduled  if  the  condition  is  true  at  the  time  the 
source  event  is  activated. 

Event  Parameters 

A  list  of  data  types  or  classes’  name  in 
parenthesis 

( int,  boolean) 

Input  parameters  types  for  the  execution  of  the  event  action. 

Edge  Arguments 

Eist  of  argument’s  names  in  parenthesis 

(n ,  is  Ready) 

Arguments  for  the  target  event.  Must  match  event  parameters. 
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B,  EG  EXAMPLE:  QUEUING  SYSTEM 

The  model  in  the  figure  simulates  a  queuing  proeess,  sueh  as  supermarket  eashier  line  or  bank  teller  line,  in  whieh  an  “arrival” 
oecurs  every  certain  time  “ta”  (a  value  draw  from  a  particular  distribution);  there  are  “S”  servers  that  can  only  attend  one  arrival  at  a  time. 
The  service  times  of  the  servers  are  “ts”  (values  draw  from  other  particular  distribution).  When  the  servers  are  busy  (s=0),  the  pending 
arrival  has  to  wait  in  a  FIFO  queue. 


Parameters: 

State  Variables 

Events  /  Actions 

{ta}  .-  inter  arrival  time  (sequence  of  random  vars) 

q  .-  number  in  the  queue  (init  val  =  0) 

Run  /  nothing 

{ts}  .-  inter  service  time  (sequence  of  random  vars  ) 

s  .-  number  of  available  servers. (init  val  =  S) 

Arrival  /  increment  q  and  n  by  one 

maxCustomers  .-  max  number  of  customers. 

n  .-  total  number  of  arrivals  (init  val  =  0) 

Start  Service  /  decrement  q  and  s  by  one 

S  =  total  number  of  servers. 

End  Service  /  increment  s  by  one 

The  simulation  starts  with  the  event  "Run"  that  schedules  an  Arrival  event  in  ta  units  of  time  (u.t).  At  time  ta  the  event  Arrival  is 
activated;  q++  and  n++  are  execute;  if  n  is  less  than  maxCutomers  a  new  Arrival  event  is  schedule  in  ta  u.t.;  if  s  is  greater  of  zero,  that  is,  at 
least  one  server  is  idle,  a  Start  Service  event  is  schedule  with  a  zero  delay.  When  Start  Service  event  is  activated,  it  executes  its  actions  and 
schedules  an  End  Service  event  in  ts  u.t.  End  Service  increments  number  of  servers  idle  and  schedules  a  new  Start  Service  event  if  the 
queue  is  not  empty. 
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APPENDIX  B.  EGGDT  REQUIREMENTS 


A,  FUNCTIONALITY  REQUIREMENTS 

The  functionality  requirements  are  broken  down  in  the  list  below  by  reference 
number  and  function. 


RF.Ol^ 

EG  Graphic  Tool 

RF.01.01 

Basic  Functionality 

RF.01.01.01 

Allow  the  user  to  start  the  EGGDT. 

RF.01.01.02 

Allow  the  user  to  Undo  and  Redo  Actions. 

RF.01.01.03 

Allow  the  user  to  Copy  and  Paste. 

RF.01.02 

EG  Manipulation 

RF.01.02.01 

Allow  the  user  to  Create  a  New  EG 

RF.Ol. 02.02 

Allow  the  user  to  Retrieve  an  EG 

RF.Ol. 02.03 

Allow  the  user  to  Save  an  EG 

RF.Ol. 02.04 

Allow  the  user  to  Print  an  EG 

RF.01.02 

Event  Manipulation 

RF.01.03.01 

Allow  the  user  to  Create  an  Event 

RF.01.03.02 

Allow  the  user  to  Delete  an  Event 

RF.01.03.03 

Delete  related  Scheduling  edges  when  Event  Source  or  Target  is 
deleted 

RF.01.03.04 

Allow  the  user  to  Specify  Events  Names 

RF.01.03.05 

Check  Events  Names  are  different 

RF.01.03.06 

Allow  the  user  to  Specify  Event  Input  Parameters 

RF.01.03.07 

Allow  the  user  to  Specify  Event  Actions 

RF.01.03.08 

Check  Event  Actions  refer  only  to  State  Variables 

RF.01.03.09 

Allow  the  user  to  Modify  Events  Names 

^  RF  stands  for  Functionality  Requirement 
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RF.01.03.11 

Allow  the  user  to  Modify  Event  Input  Parameters 

RF.01.03.12 

Allow  the  user  to  Modify  Event  Actions 

RF.01.03.13 

Check  completeness  of  events  after  modification 

RF.01.04 

Edge  Manipulation 

RF.01.04.01 

Allow  the  user  to  Create  Scheduling  Edges 

RF.Ol. 04.02 

Allow  the  user  to  Create  Canceling  Edges 

RF.Ol. 04.03 

Check  Edges  have  a  Source  Event  and  a  Target  Event  (they  can 
refer  to  the  same  Event  for  Scheduling  Edges,  but  not  for  Canceling 
Edges) 

RF.Ol. 04.04 

Allow  the  user  to  Delete  Scheduling  Edges 

RF.Ol. 04.05 

Allow  the  user  to  Delete  Canceling  Edges 

RF.Ol. 04.06 

Allow  the  user  to  Specify  the  Scheduling  Delay  Time.  Default 
delay  time  is  zero. 

RF.Ol. 04.07 

Allow  the  user  to  Specify  Edge  Arguments. 

RF.Ol. 04.08 

Check  Edge  Arguments  match  Target  Event  Input  Parameters 

RF.Ol. 04.09 

Allow  the  user  to  Specify  Edge  Conditions 

RF.01.04.10 

Allow  the  user  to  Modify  the  Scheduling  Delay  Time 

RF.Ol. 04. 11 

Allow  the  user  to  Modify  Edge  Arguments. 

RF.01.04.12 

Allow  the  user  to  Modify  Edge  Conditions 

RF.01.04.13 

Check  Edge  completeness  after  modification 

RF.Ol. 05 

Simulation  Entity  Parameters  and  Entity  State  Variables 
Manipulation 

RF.01.05.01 

Allow  the  user  to  Define  Simulation  Parameters;  A  name  has  to  be 
provided 

RF.01.05.02 

Allow  the  user  to  Define  State  Variables;  A  name  has  to  be 
provided 

RF.01.05.03 

Check  Names  and  Acronyms  of  Simulation  Parameters  and  State 
Variables  are  all  different 

RF.01.05.04 

Allow  the  user  to  Delete  Simulation  Parameters 

RF.01.05.05 

Allow  the  user  to  Delete  State  Variables 
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RF.01.05.06 

Report  any  expression  (Edge  Condition,  Edge  Argument  or  Event 
Aetion)  that  is  invalidated  when  a  Simulation  Parameters  or  State 
Variables  is  deleted. 

RF.01.05.07 

Allow  the  user  to  speeify  Simulation  Parameters  and  State 
Variables  Aeronyms 

RF.01.05.08 

Allow  the  user  to  speeify  Simulation  Parameters  and  State 
Variables  Types 

RF.01.05.09 

Allow  the  user  to  speeify  Simulation  Parameters  and  State 
Variables  Initial  Values 

RF.01.05.10 

Allow  the  user  to  delete  Simulation  Parameters  and  State 
Variables  Aeronyms 

RF.01.05.11 

Allow  the  user  to  delete  Simulation  Parameters  and  State 
Variables  Types 

RF.01.05.12 

Allow  the  user  to  delete  Simulation  Parameters  and  State 
Variables  Initial  Values 

RF.01.05.13 

Report  any  expression  (Edge  Condition,  Edge  Argument  or  Event 
Aetion)  that  is  modified  when  a  Simulation  Parameters  or  State 
Variables  property  is  modified  or  deleted. 

RF,02 

EG  design  Acceptance 

RF.02.01 

Configuration  Management 

RF.02.01.01 

Mark  EC’s  as  Configuration  Items 

RF.02.01. 02 

Keep  traek  of  EG  versions 

RF,03 

Code  Generator 

RF.03.01 

Manual  Code  Insertion 

RF.03.01.01 

Allow  the  user  to  write  the  Java  eode  of  the  Events  (“do” 
methods) 

RF.03.02 

Automatic  Code  Generation 

RF.03.02.01 

Generate  elass  skeleton  for  the  EG 

RF.03.02.02 

Generate  “get”  and  “set”  methods  for  the  simulation  parameters 

RF.03.02.03 

Generate  “get”  (no  “set”)  methods  for  the  simulation  State 
Variables 
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RF.03.02.04 

Generate  “do”  methods  skeleton  from  Events 

RF.03.02.05 

Generate  “waitDelay”  instruetions  inside  “do”  methods  of 

Souree  Events  from  Seheduling  Edges  Delay  Times 

RF.03.02.06 

Generate  “if  bloeks”  inside  “do”  methods  of  Souree  Events  from 
Edges  Conditions 

RF.03.02.07 

Generate  “interrupt”  instructions  (with  argument  the  Target 
Event)  inside  Source  Events  from  Canceling  Edges. 

RF.03.02.08 

Paste  the  code  of  the  class  -  including  the  manually  introduced  — 
and  generate  an  ASCII  file  with  name  <EGname.java> 

RF.04 

Code  Acceptance 

RF.04.01 

Configuration  Management 

RF.04.01 

Keep  track  of  Code  versions  related  to  the  EG  versions 
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B,  GUI  REQUIREMENTS  (TASK  ANALYSIS) 


The  following  table  specifies  the  interface  tasks  that  the  user  has  to  be  provided 
by  the  tool. 


RLOl’ 

EG  Graphic  Tool 

Rl.Ol.Ol 

Basic  Tasks 

Rl.Ol.Ol.Ol 

Provide  the  user  with  an  interface  to  “Start”  the  tool 

RI.01.01.02 

Provide  the  user  with  an  interface  to  “Exit”  the  tool 

RI.01.01.03 

Provide  the  user  with  an  interface  to  “Undo”  and  “Redo”  actions 

RI.01.01.04 

Provide  the  user  with  an  interface  to  “Copy”  and  “Paste” 

RI.01.01.05 

Provide  the  user  with  configurable  “Drawing  Grid” 

RI.01.01.06 

Provide  the  user  with  an  interface  for  “Configuration 
Management” 

RI.01.02 

EG  Manipulation  Tasks 

RI.01.02.01 

Provide  the  user  with  an  interface  to  “Create  New  EG” 

RI.Ol. 02.02 

Provide  the  user  with  an  interface  to  “Retrieve  EG” 

RI.Ol. 02.03 

Provide  the  user  with  an  interface  to  “Save  EG” 

RI.Ol. 02.04 

Provide  the  user  with  an  interface  to  “Print  EG” 

RI.Ol. 02.05 

Provide  the  user  with  a  resizable  “Drawing  Area” 

RI.Ol. 02.06 

Provide  the  user  with  an  interface  to  “Eayout  the  EG”  that 
optimizes  the  layout  of  Events  and  Edges  figures 

RI.01.02 

Event  Manipulation  Tasks 

RI.01.03.01 

Provide  the  user  with  an  interface  to  “Create  Event” 

RI.01.03.02 

Draw  events  in  the  specified  screen’s  position 

RI.01.03.03 

Provide  the  user  with  an  interface  to  “Delete  Event” 

RI.01.03.04 

Delete  form  screen  related  Scheduling  Edges  when  their  Source  or 
Target  Event  is  deleted 

RI.01.03.05 

Provide  the  user  with  an  interface  to  “Modify  Event  Position” 

^  RI  stands  for  Interface  Requirement 
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RI.01.03.06 

Provide  the  user  with  an  interfaee  to  “Name  Event” 

RI.01.03.07 

Provide  the  user  with  an  interfaee  to  “Define  Event  Input 
Parameters” 

RI.01.03.08 

Provide  the  user  with  an  interfaee  to  “Define  Event  Aetions” 

RI.01.03.09 

Provide  the  user  with  an  interfaee  to  “Define  priority  of  Events” 

Rl.01.04 

Edge  Manipulation  Tasks 

RI.01.04.01 

Provide  the  user  with  an  interfaee  to  “Create  Seheduling  Edge” 

RI.Ol. 04.02 

Provide  the  user  with  an  interfaee  to  “Create  Caneeling  Edge” 

RI.Ol. 04.03 

Draw  Edges  on  the  sereen  eonneeting  them  to  their  eorresponding 
Souree  and  Target  Events. 

RI.Ol. 04.04 

Provide  the  user  with  an  interface  to  “Modify  Edge  Shape” 

RI.Ol. 04.05 

Provide  the  user  with  an  interface  to  “Modify  Edge’s  Source  or 
Target  Event” 

RI.Ol. 04.06 

Provide  the  user  with  an  interface  to  “Specify  Scheduling  Edge 
Delay  Time” 

RI.Ol. 04.07 

Provide  the  user  with  an  interface  to  “Specify  Edge  Arguments” 

RI.Ol. 04.08 

Provide  the  user  with  an  interface  to  “Specify  Edge  Conditions” 

RI.Ol. 04.09 

Provide  the  user  with  an  interface  to  “Delete  Edge” 

RI.Ol. 05 

Simulation  Entity  Parameters  and  Entity  State  Variables 
Manipulation  Tasks 

RI.01.05.01 

Provide  the  user  with  an  interface  to  “Define  Simulation  Entity 
Parameter” 

RI.01.05.02 

Provide  the  user  with  an  interface  to  “Define  Simulation  Entity 
State  Variable” 

RI.01.05.03 

Provide  the  user  with  an  interface  to  “Delete  parameter  or  State 
Variable” 

RI.01.05.04 

Provide  the  user  with  an  interface  to  specify  Acronyms,  Types, 
and  Initial  Values  for  Parameters  and  State  Variables. 

RI.01.05.05 

Show  Parameters  and  State  Variables  in  screen  in  an  organized 
way 

RI.01.05.06 

Provide  the  user  with  an  interface  to  inspect  Parameters  and  State 
Variables  cross  references  with  Events  and  Edges 
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RI.02 

Code  Generator  Tasks 

RI.02.01 

Provide  the  user  with  an  interface  to  “Generate  Java  Code” 

RI.02.02 

Provide  the  user  with  an  interface  to  “Edit  Code” 
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APPENDIX  C.  FIRST  EXPERIMENT  FORMULARIES  AND  DATA 


A.  TASK  LIST  (PARTICIPANT’S  VERSION) 


A.  Describe  the  menus  and  buttons. 

B.  Create  events  :  “Run”,  “Arrival”  and  “Start”. 

Bl.  Delete  event  “Start”. 

C.  Create  an  Edge  between  events  “Run”  and  “Arrival”  with  delay  time  “ta”. 

D.  Create  an  Event  named  “Eeave”. 

E.  Create  an  edge  between  events  “Arrival”  and  “Eeave”,  with  delay  time  “tl”  and  condition 
“finishes  =  =  true”. 

E.  Modify  the  Event  Graph  layout  by  aligning  all  events  horizontally,  with  edges  as  straight 
lines  and  in  the  following  order  from  left:  Run-Arrival-Eeave. 

G.  Rename  event  “Run”  to  “Start”. 

H.  Delete  event  “Eeave”. 

I.  Create  an  edge  between  “Arrival”  and  “Start”  with  condition 

“lamEedUp  =  =  true”  (  now  there  is  an  edge  from  “Start”  to  “Arrival”  and  other  form 
“Arrival”  to  “Start”). 

J.  Modify  position  of  Edge  from  “Start”  to  “Arrival”  with  delay  time  “ta”  so  that  it  connects 
to  events  vertically  on  top  and  then  extends  horizontally  between  them.  (Edge  to  look  like 

it,  99\ 

I - 1  )• 

K.  Modify  position  of  Edge  from  “Start”  to  “Arrival”  with  delay  time  “tl”  so  that  it  connects 

to  events  vertically  on  bottom  and  then  extends  horizontally  between  them,  (edge  to  look 
like  “  I - 1  ”). 

E.  Delete  Event  “Arrival”. 

M.  Create  an  Event  named  “AlmostDone”. 

N.  Create  an  edge  between  ’’AlmostDone”.  and  “Start”  with  condition 
“lamHavingEun  =  =  true”. 
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B.  TASK  LIST  (EVALUATOR’S  VERSION) 

Benchmark  1  (measure  task  performance): 

A.  Describe  the  menus  and  buttons. 

B.  Create  events  :  “Run”  and  “Arrival”. 

Intervening  nonbenchmark  tasks: 

Bl.  Delete  event  “Start”. 

Benchmark  2  (measure  t  ask  performance  time): 

C.  Create  an  Edge  between  events  “Run”  and  “Arrival”  with  delay  time  “ta”. 

Intervening  nonbenchmark  tasks: 

D.  Create  an  Event  named  “Eeave”. 

E.  Create  an  edge  between  events  “Arrival”  and  “Eeave”,  with  delay  time  “tl”  and  condition 
“finishes  =  =  true”. 

Benchmark  3  (measure  task  performance  time) 

E.  Modify  the  Event  Graph  layout  by  aligning  all  events  horizontally,  with  edges  as  straight 
lines  and  in  the  following  order  from  left:  Run-Arrival-Eeave. 

Benchmark  4  (measure  task  performance  time) 

G.  Rename  event  “Run”  to  “Start”. 

Intervening  nonbenchmark  tasks: 

H.  Delete  event  “Eeave”. 

I.  Create  an  edge  between  “Arrival”  and  “Start”  with  condition 

“lamEedUp  =  =  true”.  ”  (  now  there  is  an  edge  from  “Start”  to  “Arrival”  and  other  form 
“Arrival”  to  “Start”). 

Benchmark  5  (measure  task  performance  time) 

J.  Modify  position  of  Edge  from  “Start”  to  “Arrival”  with  delay  time  “ta”  so  that  it  connects 
to  events  vertically  on  top  and  then  extends  horizontally  between  them.  (Edge  to  look  like 

I - 1  )• 

Intervening  nonbenchmark  tasks: 

K.  Modify  position  of  Edge  from  “Start”  to  “Arrival”  with  delay  time  “tl”  so  that  it  connects 

to  events  vertically  on  bottom  and  then  extends  horizontally  between  them,  (edge  to  look 
like  “  I - 1  ”). 

E.  Delete  Event  “Arrival”. 

M.  Create  an  Event  named  “AlmostDone”. 

Benchmark  6  -final  task  -  (measure  task  performance  time) 
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N.  Create  an  edge  between  “AlmostDone”  and  “Start”  with  condition 
“lamHavingFun  =  =  true”. 

C.  INFORMED  CONSENT  FORM 

You  are  going  to  become  a  research  participant  for  our  evaluation  of  the  design  for  a  new 
interactive  computer  system.  This  evaluation  is  being  conducted  by  Angel  San  Jose  (OR)  and 
Paulo  Silva  (CSOl)  and  is  part  of  a  project  for  the  MV4203  -  Interactive  Computation 
Systems  and  the  thesis  of  Angel  San  Jose.  Angel  SanJose,  who  will  be  glad  to  answer  you 
any  questions  about  this  evaluation  session,  will  run  your  evaluation  session.  As  a 
participant,  you  have  some  rights,  which  are  listed  bellow. 

You  will  be  asked  to  sit  in  from  of  a  Lab  PC  and  perform  a  number  of  tasks  associated 
with  the  systems.  We  are  evaluating  the  system,  to  make  it  as  effective  and  usable  as 
possible.  We  are  not  in  any  way  evaluating  you.  We  expect  the  session  to  last  about  30 
minutes.  The  evaluator  will  be  monitoring  you  throughout  the  session  to  obtained  the 
required  feed  back  from  the  session.  Your  name  will  not  be  associated  with  any  data 
collected.  There  are  no  known  risks  associated  with  this  evaluation.  You  will  be  given  a 
written  tasks  list  and  a  small  questionnaire  at  the  end  for  you  to  express  you  opinion. 

Your  rights  as  a  participant  are  as  follows: 

1 .  You  have  the  right  to  withdraw  from  the  session  at  any  time. 

2.  At  the  end  of  you  session,  you  may  see  your  data,  if  you  so  desire.  If  you  wish  to 
withdraw  your  data  at  that  point,  inform  the  evaluator.  Otherwise,  it  might  be  impossible 
because  of  our  efforts  to  keep  anonymity. 

3.  Ensure  you  do  not  comment  this  session  with  any  other  participants  that  have  not  yet 
performed  their  session. 

Finally,  we  greatly  appreciate  your  time  and  effort  for  participating  and  helping  us  in  this 
evaluation.  Remember,  this  is  not  a  pass-fail  assessment;  instead  it  is  a  useful  contribution  to 
the  development  of  this  computer  system.  Your  signature  bellow  indicates  that  you  have  read 
and  understand  the  contents  of  this  consent  form  and  that  you  entirely  agree  to  participate. 

Do  you  have  any  questions? 

NAME _ 

SIGNATURE _ 

DATE _ 

D.  SESSION  INSTRUCTIONS 

We  thank  you  very  much  for  your  time  and  participation  in  this  experiment.  You  help 
will  enable  us  to  evaluate  this  novel  tool  that  will  help  you  creating  Event  Graphs  (EG). 

As  you  can  observe,  this  a  simple  drawing  tool  that  will  be  used  to  draw  an  EG  and  will 
be  able  to  run  in  any  platform  which  has  a  Java  Run  time  environment. 

You  have  been  designing  simulations  models  using  EGs  which  you  draw  by  hand,  and 
translated  to  Java  using  Simkit  also  manually.  The  propose  of  this  tool  is  to  allow  you  to 
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create  an  EG  and  to  automatieally  generate  the  eorresponding  Java  Class  using  the  Simkit 
library. 

The  prototype  only  permits  to  draw,  modify  and  delete  events  and  edges  — seheduling  but 
not  eaneeling  nor  self  edges.  The  rest  of  the  funetions  of  the  final  tool  are  not  yet 
implemented.  Partieularly  you  ean:  draw  and  name  an  event,  modify  its  position  in  the 
sereen,  delete  an  event,  ereate  an  edge  between  to  events  -  arrow  heads  are  not  yet 
implemented,  introduee  the  edge  properties  (delay  times,  arguments  and  eonditions),  modify 
the  shape  of  an  edge  and  delete  edges. 

A  different  notation,  with  respeet  to  the  one  learned  in  the  Simulation  elass,  has  been 
introdueed.  The  edges  are  not  eurves  but  segments  of  straight  lines  and  the  properties  of  the 
edge  (delay  time,  arguments  and  eonditions)  have  to  be  written  in  the  same  line  with  the 
format:  delay  time  (argument  list)  [condition  list]. 

During  this  evaluation,  you  will  be  asked  to  perform  a  number  of  speeifie  tasks  using  this 
system.  Then  we  will  you  give  you  some  free  time  to  play  around  with  the  tool,  exploring  it 
the  way  you  want.  We  then  will  ask  you  to  perform  a  few  more  speeifie  tasks  and  finally. 

The  list  of  tasks  will  be  given  to  you  in  written  format  and  we  ask  you  to  read  eaeh  task 
aloud  and  to  make  sure  you  understand  it  before  you  begin.  In  order  to  obtain  the  best 
possible  feedbaek  from  you  partieipation,  we  will  probably  be  timing  how  well  the  system  is 
helping  you  on  those  tasks.  We  therefore  ask  you  to  work  through  eaeh  individual  task 
without  stopping.  When  you  are  done  with  one  task  you  ean  then  relax  before  going  into  the 
next  one. 

While  you  are  doing  your  tasks,  we  ask  you  to  think  aloud.  By  doing  that  we  ean 
therefore  observe  if  the  system  is  helping  you  the  way  you  expeet  or  if  it  is  laeking  some 
behavior  or  eausing  any  diffieulties  to  your  notions.  Feel  oompletely  free  to  oomment  all  bad 
and  good  aspeots  of  the  interfaoe  as  you  go  along.  The  more  oomments  you  tell  us  the  more 
helpful  will  your  partieipation  be. 

This  is  by  no  means  an  evaluation  of  your  self.  Rather,  you  are  only  helping  us  evaluating 
this  tool  so  that  we  ean  improve  it  to  meet  your  best  expeotations  for  a  tool  like  this. 

When  you  are  done  with  the  task  list,  we  will  ask  you  to  eomplete  a  short  questionnaire  to 
rate  the  system  and  to  express  your  overall  opinion  about  it. 

We  expeet  this  session  to  last  30  minutes  in  total. 

Before  you  start,  do  you  have  any  question? 

E.  QUESTIONNAIRE  FOR  THE  USER  INTERFACE  SATISFACTION 

For  eaeh  of  the  following  questions,  fill  in  0-9  or  leave  blank  if  question  is  not  applieable) 
Skip  question  if  not  applieable 


REMEMBER,  YOU  ARE  NOT  EVALUATING  US  BUT  THE  TOOL. 
SINCERITY  IS  APPRECIATED 
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1.  Overall  Reaction  to  the  Event  Graph  Design  Tool 

terrible  0123456789  wonderful 

difficult  0123456789  easy 

frustrating  0123456789  satisfying 

dull  0123456789  stimulating 

rigid  0123456789  flexible 

2.  Screen 

Characters  on  the  eomputer  sereen 

hard  to  read  0123456789  easy  to  read 

Tool  bar  with  buttons 

hard  to  read  0123456789  easy  to  read 

Organization  of  information  on  sereen 

confusing  0123456789  very  clear 

3.  Terminology  and  System  Information 

Computer  terminology  is  related  to  the  task  you  are  doing 

never  0123456789  always 

Menu  Items  and  Tool  bar  with  buttons  funetion  identifieation 
difficult  to  identify  0123456789  easy  to  read 


4.  Learning 

Learning  to  operate  the  system 

difficult  0123456789 

Exploring  new  features  by  trial  and  error 

difficult  0123456789 

Remembering  names  and  use  of  eommands 

difficult  0123456789 


easy 


easy 


easy 


Tasks  ean  be  performed  in  a  straightforward  manner 


never  0123456789  always 


5,  System  Capabilities 

System  speed 

too  slow  0123456789  fast  enough 
System  reliability 
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unreliable  0123456789  reliable 

Correcting  your  mistakes 

difficult  0123456789  easy 

Experienced  and  inexperienced  users'  needs  are  taken  into  consideration 
never  0123456789  always 


COMMENTS 
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F,  USABILITY  SPECIFICATION  TABLE 


Usability 

Attribute 

Measuring 

Instrument 

Value  to  be 
Measured 

Current 

Level 

(Secs) 

Worst 

Acceptable 

Level(Secs) 

Planned 

Target 

Level 

(Secs) 

Best 

Possible 

Level 

(Secs) 

Observed  Results 
(Secs) 

Initial 

“Create  an  Event” 

Eength  of  time  to 

60  (using 

60 

50 

20 

PL. 5  =  80  /  57  /49  /43  / 

performance 

task  per 

successfully 

standard 

47 

Benchmark  1 

create  an  event 

graphical 

Mean  =  55.2 

tool) 

5  (by  hand) 

Stdv  =  14.77 

Initial 

“Create  an  Edge” 

Eength  of  time  to 

120 

120 

60 

20 

PI. .5  =  113  /  120  /  120  / 

performance 

task  per 

successfully 

5 

47/17 

Benchmark  2 

create  an  edge 

Mean  =  83.4 

Stdv  =  48.19 

Initial 

“Modify  EG 

Eength  of  time  to 

60 

60 

30 

10 

PI. .5  =  60  /20  /  99  /  56  / 

performance 

layout”  task  per 

successfully 

120 

50 

Benchmark  3 

modify  EG 

Mean  =  57.0 

layout 

Stdv  =  28.25 

Initial 

“Rename  Event” 

Eength  of  time  to 

20 

20 

10 

5 

PI. .5  =  10/9/2/4/45 
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Usability 

Attribute 

Measuring 

Instrument 

Value  to  be 
Measured 

Current 

Level 

(Secs) 

Worst 

Acceptable 

Level(Secs) 

Planned 

Target 

Level 

(Secs) 

Best 

Possible 

Level 

(Secs) 

Observed  Results 
(Secs) 

performance 

task  per 

successfully 

20 

Mean  =14 

Benchmark  4 

rename  and 

Event 

Stdv  =  7.65 

Initial 

“Modify  Edge” 

Eength  of  time  to 

60 

60 

20 

40 

performance 

task  per 

successfully 

60 

Benchmark  5 

modify  Edge 

Leamability 

Repeat  “Create  an 

Eength  of  time  to 

120 

120 

60 

20 

P1..5  =  37  /  21  /  37  /  28  / 

Edge”  task  at  the 

successfully 

5 

33 

end  of  the  session 

create  an  edge 

Mean  =  31.2 

per  Benchmark  6 

Stdv  =  6.80 

First 

Participant 

Average  score 

Not  know 

5 

7 

9 

PI. .5  =  7.0  /  6.9/  8.3  11.5 

impression 

responds  to  QUIS 

(From  0  to  9) 

but  low 

/8.1 

Questionnaire 

Grand  Mean  =  7.54 
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APPENDIX  D.  SECOND  EXPERIMENT  FORMS  AND  DATA 


A,  PARTICIPANT  CONSENT  FORM 

1.  Introduction.  You  are  invited  to  partieipate  in  a  eomparative  experiment  of  ares 
modalities  to  be  used  in  the  Event  Graph  Graphical  Design  Tool  (EGGDT)  .  With 
information  gathered  from  you  and  other  participants  we  will  develop  a  tool  for 
designing  Event  Graph  for  simulation  models.  We  ask  you  to  read  and  sign  this  form 
indicating  that  you  agree  to  be  in  the  study.  Please  ask  any  questions  you  may  have 
before  signing. 

2.  Background  Information,  This  experiment  is  part  of  the  thesis  research  currently 
carried  out  by  Angel  San  Jose  (OA  curriculum)  and  the  final  project  of  the  team  A  for 
the  OA-3401  course. 

3.  Procedures,  If  you  agree  to  participate  in  this  study,  the  researcher  will  launch  a 
computer  application.  Every  step  of  the  experiment  will  be  explained  in  detail  in  the 
different  windows.  The  researcher  will  be  available  to  help  you  in  the  preliminary 
phases.  You  are  expected  to  perform  the  benchmark  tasks  by  yourself.  In  any  case  if 
you  think  you  cannot  complete  the  task,  ask  for  assistance. 

4.  Risks  and  Benefits,  This  research  involves  no  risks  or  discomforts  greater  then  those 
encountered  in  ordinary  computer  work.  The  final  product  will  benefit  OR  analysts. 

5.  Compensation,  No  tangible  reward  will  be  given.  You  will  see  your  results  at  the 
conclusion  of  the  experiment. 

6.  Confidentiality,  The  records  of  this  study  will  be  kept  confidential.  No  information 
will  be  publicly  accessible  which  could  identify  you  as  a  participant. 

7.  Voluntary  Nature  of  the  Study,  If  you  agree  to  participate,  you  are  free  to  withdraw 
from  the  study  at  any  time  without  prejudice.  You  will  be  provided  a  copy  of  this  form 
for  your  records. 

8.  Points  of  Contact,  If  you  have  any  further  questions  or  comments  after  the  completion 
of  the  study,  you  may  contact  the  research  supervisor.  Dr.  Nita  Miller  (telephone  656- 
2281)  or  the  Principal  Investigator  Angel  San  Jose  (aesanjos@nps.navy.mil). 

9.  Statement  of  Consent.  I  have  read  the  above  information.  I  have  asked  all  questions 
and  have  had  my  questions  answered.  I  agree  to  participate  in  this  study. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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B,  MINIMAL  RISK  CONSENT  STATEMENT 


Participant:  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT 
IN:  Arcs  modalities  to  be  used  in  the  Event  Graph  Graphical 
Design  Tool  (EGGDT)  . 


1.  I  have  read,  understand  and  been  provided  "Information  for  Participants"  that  provides  the 
details  of  the  below  acknowledgments. 

2.  I  understand  that  this  project  involves  research.  An  explanation  of  the  purposes  of  the 
research,  a  description  of  procedures  to  be  used,  identification  of  experimental  procedures, 
and  the  extended  duration  of  my  participation  have  been  provided  to  me. 

3.  I  understand  that  this  project  does  not  involve  risk.  I  have  been  informed  of  any  reasonably 
foreseeable  risks  or  discomforts  to  me. 

4.  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be  expected  from 
the  research. 

5.  I  have  signed  a  statement  describing  the  extent  to  which  confidentiality  of  records  identifying 
me  will  be  maintained. 

6.  I  have  been  informed  of  any  compensation  and/or  medical  treatments  available  if  injury 
occurs  and  if  so,  what  they  consist  of,  or  where  further  information  may  be  obtained. 

7.  I  understand  that  my  participation  in  this  project  is  voluntary,  refusal  to  participate  will 
involve  no  penalty  or  loss  of  benefits  to  which  I  am  otherwise  entitled.  I  also  understand  that 
I  may  discontinue  participation  at  any  time  without  penalty  or  loss  of  benefits  to  which  I  am 
otherwise  entitled. 

8.  I  understand  that  the  individual  to  contact  should  I  need  answers  to  pertinent  questions  about 
the  research  is  Professor  Nita  Miller  or  the  Principal  Investigator  Angel  San  Jose,  and 
about  my  rights  as  a  research  participant  or  concerning  a  research  related  injury  is  the 
Operational  Research  Department  Chairman  James  Eagle.  A  full  and  responsive  discussion 
of  the  elements  of  this  project  and  my  consent  has  taken  place. 


Signature  of  Principal  Investigator 

Date 

Signature  of  Volunteer 

Date 

Signature  of  Witness 

Date 
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C.  PRIVACY  ACT  STATEMENT 

NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 
PRIVACY  ACT  STATEMENT 


1 .  Authority:  Naval  Instruction 

2.  Purpose:  Comparison  between  arcs  modalities  to  be  used 
in  the  Event  Graph  Graphical  Design  Tool  (EGGDT)  . 

3.  Use:  The  ares  modality  seleeted  will  be  used  in  the  EGGDT  to  allow  the  user  to 
draw  edges  connecting  events. 

4.  Disclosure/Confidentiality: 

a.  I  have  been  assured  that  my  privacy  will  be  safeguarded.  I  will  be  assigned  a 
eontrol  or  code  number,  which  thereafter  will  be  the  only  identifying  entry  on 
any  of  the  research  records.  The  Prineipal  Investigator  will  maintain  the  eross- 
reference  between  name  and  control  number.  It  will  be  deeoded  only  when 
beneficial  to  me  or  if  some  circumstanees,  whieh  is  not  apparent  at  this  time, 
would  make  it  elear  that  deeoding  would  enhance  the  value  of  the  researeh  data. 
In  all  cases,  the  provisions  of  the  Privacy  Act  Statement  will  be  honored. 

b.  I  understand  that  a  record  of  the  information  eontained  in  this  Consent  Statement 
or  derived  from  the  experiment  deseribed  herein  will  be  retained  permanently  at 
the  Naval  Postgraduate  School  or  by  higher  authority.  I  voluntarily  agree  to  its 
diselosure  to  agencies  or  individuals  indicated  in  paragraph  3  and  I  have  been 
informed  that  failure  to  agree  to  sueh  diselosure  may  negate  the  purpose  for 
whieh  the  experiment  was  conducted. 

c.  I  also  understand  that  disclosure  of  the  requested  information,  ineluding  my 
Soeial  Security  Number,  is  voluntary. 


Signature  of  Volunteer  Name,  Grade/Rank  (if  applicable)  DOB  SSN  Date 


Signature  of  Witness  Date 
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D, 


COMPUTER  PROFICIENCY  SURVEY 


Participant  ID  Number _  Computer _ 

Test  A  Time _  Test  A  Errors  _  First  ? 

Test  B  Time  Test  B  Errors  First  ? 


1 .  How  many  years  of  basic  computer  use  do  you  have?  _ 

2.  What  is  your  pereeived  eomputer  experienee  level? 

a.  None 

b.  A  little  experienee 

c.  Moderate 

d.  I  can  make  eomputers  do  what  a  I  want 

e.  Expert 

3.  Do  you  have  eomputer  at  home?  _ 

4.  Rate  your  profieieney  in  graphieal  or  presentation  software  (i.e.  Powerpoint, 
Harvard  Graphies) 

a.  None 

b.  I  don’t  know  that  mueh  about  these  software  titles 

c.  Moderate  profieieney 

d.  I  ean  build  a  strong  presentation 

e.  I  ean  make  the  sereen  danee  a  fine  jig 

5.  Rate  your  eomputer  mouse  agility. 

a.  Mouse?  I  didn’t  think  that  we  were  testing  hygienic  products  on  lab 
animals! 

b.  Poor 

c.  Average 

d.  Good 

e.  Exeellent 

6.  Whieh  statement  best  deseribes  your  feelings  towards  eomputers? 

a.  Someone  get  me  a  sledgehammer. 

b.  I  could  take  them  or  leave  them. 

e.  I  think  they  ean  do  some  pretty  eool  stuff. 

d.  I  like  eomputers. 

e.  I  would  marry  one. 

Part  II  -  Are  Preferenee 

1 .  Whieh  of  the  two  modes  of  are  manipulation  in  the  experiment  do  you  prefer? 

a.  Direet  draw 

b.  Right  Angle 

2.  Have  you  partieipated  in  an  experiment  related  with  this  EGGDT  program? 


Part  III  -  Demographies 
1 .  How  old  are  you?  _ 
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2.  Which  is  your  dominant  hand  or  are  you  ambidextrous?  _ 

3.  What  is  your  gender?  _ 

4.  Do  you  prefer  to  use  a  computer  mouse  with  your  right  or  left  hand? 


5.  You  ean  configure  a  eomputer  mouse  to  be  used  either  “left-handed”  or 

“right-handed”.  Whieh  configuration  do  you  use?  _ 

Part  IV  —  Comments 

Do  you  have  any  recommendations  for  improving  the  arc  manipulation  methods? 
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E.  DATA 


Symbol 

Data  Table  Name 

Explanation 

# 

# 

Partieipant’s  order 

ID 

ID 

Combination  of  #  and  Controller 

Controller 

Controller 

Initial  of  the  experiment’s  eontroller 

Lab 

Lab 

Laboratory  (Human  Faetors  or  Loosely  Coupled  Components) 

VH.Time 

VH.Time 

Time  to  perform  VH  treatment 

F.Time 

F.Time 

Time  to  perform  F  treatment 

Dif.VH.F 

Time.Diff.VH.F 

VH.Time  -  F.Time 

E.VH 

E.VH 

Errors  in  VH  treatment 

E.F 

E.F 

Errors  in  F  treatment 

Order 

Order 

Order  of  treatments  (VH/F  or  FA^H) 

Exp 

Exper.years 

Computer  experienee 

Level 

Level 

Self  expressed  eomputer  level 

Home  PC 

Home.PC 

Partieipant  has  PC  at  home 

Graph  Exper 

Graphieal 

Self  expressed  experienee  with  graphieal  environments 

Mouse  DEX 

Mouse. DEX 

Self  expressed  mouse  dexterity 

PC  Emp 

PC.Emp 

Feelings  towards  eomputers 

Pref 

Pref 

Self  Expressed  preferenee  VH  or  F 

98 


Age 

Age 

Partieipant’s  age 

RH  domi 

RH. Dominant 

Right  hand  dominant 

RH  Mouse 

RH.Mouse.User 

Right  hand  mouse  user 

Mouseeonfi 

RH.Mouse.Config 

Right  hand  mouse  eonfiguration 

Gender 

Male. Gender 

Partieipant’s  gender 

APPENDIX  E.  FINAL  EXPERIMENT  FORMS  AND  DATA 


A,  PARTICIPANT  CONSENT  FORM 

1.  Introduction.  You  are  invited  to  participate  in  an  evaluation  experiment  of  the 
Event  Graph  Graphical  Design  Tool  (EGGDT)  .  With  information  gathered  from 
you  and  other  participants  I  will  improve  my  tool  for  designing  Event  Graph  for 
simulation  models.  I  ask  you  to  read  and  sign  this  form  indicating  that  you  agree  to 
be  in  the  study.  Please  ask  any  questions  you  may  have  before  signing. 

2.  Background  Information,  This  experiment  is  part  of  the  thesis  research  currently 
carried  out  by  Angel  San  Jose  (OA  curriculum). 

3.  Procedures.  If  you  agree  to  participate  in  this  study,  you  will  be  provided  with  a 
computer  in  which  the  EGGDT  application  will  be  install.  I  will  explain  the  use  and 
possibilities  of  the  current  EGGDT  prototype  version.  You  will  have  the 
opportunity  to  follow  me  in  a  tour  through  the  application  (this  step  will  take  a 
quarter  of  hour  approximately).  After  this  you  will  be  able  to  play  with  the  tool  by 
yourself  I  will  be  available  for  answering  question  in  any  moment.  As  a  last  task  I 
will  provide  a  questionnaire  about  the  application  for  you  to  fdl. 

4.  Risks  and  Benefits.  This  research  involves  no  risks  or  discomforts  greater  then 
those  encountered  in  ordinary  computer  work.  The  final  product  will  benefit  OR 
analysts.  In  the  short  term  it  will  benefit  particularly  those  of  you  who  are  or  will  be 
involved  in  simulation  thesis  researches. 

5.  Compensation,  No  tangible  reward  will  be  given 

6.  Confidentiality,  The  records  of  this  study  will  be  kept  confidential.  No 
information  will  be  publicly  accessible  which  could  identify  you  as  a  participant. 

7.  Voluntary  Nature  of  the  Study,  If  you  agree  to  participate,  you  are  free  to 
withdraw  from  the  study  at  any  time  without  prejudice  . 

8.  Points  of  Contact.  If  you  have  any  further  questions  or  comments  after  the 
completion  of  the  study,  you  may  contact  the  research  supervisor.  Dr.  Nita  Miller 
(telephone  656-2281)  or  the  Principal  Investigator  Angel  San  Jose 
(aesanjos@nps.navy.mil). 

9.  Statement  of  Consent,  I  have  read  the  above  information.  I  have  asked  all 
questions  and  have  had  my  questions  answered.  I  agree  to  participate  in  this  study. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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B,  MINIMAL  RISK  CONSENT  STATEMENT 

Participant;  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT 
IN  the  evaluation  of  the  Event  Graph  Graphieal  Design  Tool  (EGGDT)  . 


1.  I  have  read,  understand  and  been  provided  "Information  for  Partieipants"  that 
provides  the  details  of  the  below  aeknowledgments. 

2.  I  understand  that  this  projeet  involves  researeh.  An  explanation  of  the  purposes  of  the 
researeh,  a  deseription  of  proeedures  to  be  used,  identifieation  of  experimental 
proeedures,  and  the  extended  duration  of  my  partieipation  have  been  provided  to  me. 

3.  I  understand  that  this  projeet  does  not  involve  risk.  I  have  been  informed  of  any 
reasonably  foreseeable  risks  or  diseomforts  to  me. 

4.  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be 
expeeted  from  the  researeh. 

5.  I  have  signed  a  statement  deseribing  the  extent  to  whieh  eonfidentiality  of  reeords 
identifying  me  will  be  maintained. 

6.  I  have  been  informed  of  any  eompensation  and/or  medieal  treatments  available  if 
injury  oeeurs  and  if  so,  what  they  eonsist  of,  or  where  further  information  may  be 
obtained. 

7.  I  understand  that  my  partieipation  in  this  projeet  is  voluntary;  refusal  to  partieipate 
will  involve  no  penalty  or  loss  of  benefits  to  whieh  I  am  otherwise  entitled.  I  also 
understand  that  I  may  diseontinue  partieipation  at  any  time  without  penalty  or  loss  of 
benefits  to  whieh  I  am  otherwise  entitled. 

8.  I  understand  that  the  individual  to  eontaet  should  I  need  answers  to  pertinent 
questions  about  the  researeh  is  Professor  Nita  Miller  or  the  Prineipal  Investigator 
Angel  San  Jose,  and  about  my  rights  as  a  researeh  partieipant  or  eoneeming  a 
researeh  related  injury  is  the  Operational  Researeh  Department  Chairman  James 
Eagle.  A  full  and  responsive  diseussion  of  the  elements  of  this  projeet  and  my 
eonsent  has  taken  plaee. 


Signature  of  Principal  Investigator 

Date 

Signature  of  Volunteer 

Date 

Signature  of  Witness 

Date 
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C.  PRIVACY  ACT  STATEMENT 

1 .  Authority  :Naval  Instruction 

Purpose:  Evaluation  of  the  Event  Graph  Graphieal  Design  Tool  (EGGDT). 

2.  Use:  The  EGGDT  allows  the  Analysts  design  simulation  models  using  Event  Graph 
and  produce  the  corresponded  Java  code  using  the  Simulation  Kit  (Simkit) 
developed  by  the  NPS  Professor  Buss. 

3.  Disclosure/Confidentiality: 

d.  I  have  been  assured  that  my  privaey  will  be  safeguarded.  I  will  be  assigned  a 
control  or  code  number  which  thereafter  will  be  the  only  identifying  entry  on  any 
of  the  researeh  records.  The  Principal  Investigator  will  maintain  the  eross- 
referenee  between  name  and  control  number.  It  will  be  decoded  only  when 
beneficial  to  me  or  if  some  eircumstances,  which  is  not  apparent  at  this  time, 
would  make  it  clear  that  deeoding  would  enhance  the  value  of  the  researeh  data. 
In  all  eases,  the  provisions  of  the  Privaey  Act  Statement  will  be  honored. 

e.  I  understand  that  a  reeord  of  the  information  contained  in  this  Consent  Statement 
or  derived  from  the  experiment  described  herein  will  be  retained  permanently  at 
the  Naval  Postgraduate  School  or  by  higher  authority.  I  voluntarily  agree  to  its 
diselosure  to  ageneies  or  individuals  indicated  in  paragraph  3  and  I  have  been 
informed  that  failure  to  agree  to  sueh  diselosure  may  negate  the  purpose  for 
whieh  the  experiment  was  condueted. 

f.  I  also  understand  that  disclosure  of  the  requested  information,  including  my 
Social  Security  Number,  is  voluntary. 


Signature  of  Volunteer  Name,  Grade/Rank  (if  applicable)  DOB  SSN  Date 


Signature  of  Witness  Date 
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D,  COMPUTER  PROFICIENCY  SURVEY 

I.  Computer  proficiency 

A.  What  is  your  perceived  COMPUTER  level? 

1.  None. 

2.  A  little  experience 

3.  Moderate. 

4.  Good 

5.  Expert 

B.  Which  statement  best  describes  your  feelings  towards  COMPUTERS? 

1 .  Someone  get  me  a  sledgehammer. 

2.  I  could  take  them  or  leave  them 

3.  I  think  they  can  do  some  pretty  cool  stuff. 

4.  I  like  computers 

5.  I  would  marry  one. 

C.  Which  statement  best  describes  your  feelings  towards  SIMUEATION? 

1 .  I  should  have  enrolled  in  the  IT  curriculum. 

2.  I  could  take  them  or  leave  them.  I  won’t  use  it  again. 

3.  I  think  it  can  help  to  do  some  pretty  cool  stuff. 

4.  I  like  simulation 

5.  I  think  I  going  to  focus  my  professional  future  in  this  area. 

II.  Simulation  Design  tools  evaluation. 

A.  Do  you  think  this  tool,  or  tools  like  this,  can  improve  your  satisfaction  when 
developing  simulation  applications? 

1.  Yes. 

2.  No. 

3.  No  opinion. 

B.  To  develop  simulation  models,  do  you  prefer  to  use  the  methods  you  have 
been  using  so  far  or  some  kind  of  tool  like  this? 

1.  Current  tools. 

2.  Tools  like  this. 

3.  No  opinion. 

C.  After  viewing  this  presentation,  how  useful  do  you  think  these  tools  will  be  in 
the  OR  field. 

1 .  These  tools  are  not  needed. 

2.  These  tools  are  valid  but  not  needed. 

3.  These  tools  could  have  limited  utility  in  specific  applications. 

4.  These  tools  have  utility  in  a  wide  range  of  applications. 

5.  These  tools  are  a  major  step  forward  in  the  OR  field. 

D.  Assuming  you  were  skeptical  about  using  simulation  in  your  future  OR 
projects;  how  would  you  rate  your  feelings  about  simulation  after  having  seen  this 
experiment? 

1 .  My  feelings  are  unchanged;  I  probably  would  not  use  simulation. 

2.  These  tools  seem  useful  but  I  would  only  use  them  as  a  last  resort. 

3.  With  a  tool  like  this,  I  think  I  could  design  some  useful  simulations 
with  confidence. 
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4.  I  am  very  confidence  that  these  tools  will  cause  me  to  favor  simulation 
in  most  OR  projects. 

5.  I  now  believe  that  simulation  should  be  my  primary  OR  tool. 

III.  Demographics 

A.  How  old  are  you? _ 

B.  What  is  your  gender? _ 

C.  Military  branch 

1.  US  Navy. 

2.  USMC. 

3.  US  Army. 

4.  USAF. 

5.  International. 

6.  Other _ 

D.  Curriculum/quarter _ 

IV.  Comments 

YOUR  SUGGESTIONS  ARE  VERY  IMPORTANT. 

Do  you  have  any  recommendations  for  improving  the  EGGDT? 
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E.  DATA 


Num 

Session 

Level 

FeelCom 

FeelSim 

Satisfaction 

Preference 

Useful 

NewFeelSim  Age 

Gender 

Branch 

Curriculum 

Quarter 

1 

2 

4 

4 

4 

1 

2 

4 

3 

26 

M 

5 

OR 

4 

2 

2 

4 

2 

4 

1 

2 

4 

3 

43 

M 

5 

OR 

4 

3 

2 

4 

3 

3 

1 

2 

4 

3 

35 

M 

5 

OR 

4 

4 

2 

4 

4 

4 

1 

2 

4 

4 

28 

M 

5 

OR 

4 

5 

2 

3 

4 

3 

1 

2 

5 

4 

32 

M 

3 

OR 

4 

6 

2 

3 

4 

4 

1 

2 

4 

4 

26 

M 

5 

OR 

4 

7 

2 

4 

3 

3 

1 

2 

3 

3 

34 

M 

3 

OR 

4 

8 

2 

3 

3 

3 

1 

2 

4 

3 

28 

M 

1 

OR 

8 

9 

2 

4 

3 

3 

1 

2 

4 

3 

30 

M 

3 

OR 

4 

10 

2 

4 

4 

4 

1 

2 

4 

3 

32 

F 

1 

OR 

4 

11 

1 

3 

4 

4 

1 

2 

5 

4 

28 

M 

2 

OR 

8 

12 

1 

3 

4 

3 

1 

2 

4 

3 

30 

F 

1 

OR 

8 

13 

1 

3 

4 

3 

1 

2 

3 

3 

34 

M 

2 

OR 

8 

14 

1 

3 

3 

4 

1 

2 

4 

3 

40 

M 

2 

OR 

4 

15 

1 

4 

3 

3 

1 

2 

3 

3 

30 

M 

2 

OR 

4 

16 

1 

4 

4 

5 

1 

2 

4 

3 

39 

F 

1 

OR 

6 

17 

1 

3 

3 

3 

1 

2 

4 

3 

38 

M 

1 

OR 

8 

18 

1 

3 

3 

3 

1 

2 

3 

3 

32 

M 

1 

OR 

4 

19 

1 

3 

3 

3 

1 

2 

4 

4 

33 

F 

1 

OR 

8 
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F.  FIELDS  EXPLANATION 


Num 

Partieipant’s  order  number 

Session 

Two  session  took  plaee 

Level 

Partieipant’s  Computer  level 

FeelCom 

Partieipant’s  feelings  toward  eomputers 

FeelSim 

Partieipant’s  feelings  toward  simulation 

Satisfaetion 

Partieipant’s  opinion  if  IDE  tools  improve  satisfaetion 

Preferenee 

Partieipant’s  preferenee  between  old  tools  or  IDE  tools 

Useful 

Partieipant’s  opinion  in  usefulness  of  the  IDEs 

NewFeelSim  Partieipant’s  feelings  toward  Simulation  after  the  session 

Age 

Partieipant’s  age 

Gender 

Partieipant’s  gender 

Braneh 

Partieipant’s  military  braneh 

Currieulum 

Partieipant’s  eurrieulum 

Quarter 

Partieipant’s  quarter 
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